2024-10-16 13:22:38.168 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:22:38.216 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:38.218 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:38.219 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:22:38.220 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:22:38.220 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:38.221 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:22:38.231 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:22:38.231 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:22:38.231 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:22:38.231 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:22:38.231 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:22:38.231 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:38.298 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:22:38.464 INFO 4486 --- [ocp-schedules-28,9e58c06e209ddd43,7710d2236451379d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:22:38.513 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:22:38.301244, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:22:38.513 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:22:38.301244, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:22:38.553 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:22:38.301244} } 2024-10-16 13:22:38.606 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:22:38.612 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:22:38.615 INFO 4486 --- [ocp-schedules-21,161f24c2a54e98d8,689a60ae09417f83] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:22:38.702 INFO 4486 --- [ocp-schedules-9,2b23f34cf44d7bd8,06d5a0bbd95567f9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:22:38.801 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d7d3e204b4707f78,9063903d6493b36d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d7d3e204b4707f78, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:38.801 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d7d3e204b4707f78,9063903d6493b36d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d7d3e204b4707f78, duration=2 ms] 2024-10-16 13:22:38.888 INFO 4486 --- [ocp-async-8,fccedf709bf827bb,45e3974861b27350] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:39.424 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:22:39.921 INFO 4486 --- [ocp-schedules-8,ef28a83aba61d044,75abeaeb8452c721] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:22:40.005 INFO 4486 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:22:40.007 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:22:40.026 INFO 4486 --- [Check obproxy2,902479d8c59521e7,3040ad19ae8307c2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:22:40.026 INFO 4486 --- [Check obproxy2,902479d8c59521e7,3040ad19ae8307c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:22:40.028 INFO 4486 --- [Check obproxy1,3ad9a5c603736c04,41b99cff95196ec5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:22:40.028 INFO 4486 --- [Check obproxy1,3ad9a5c603736c04,41b99cff95196ec5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:22:40.031 INFO 4486 --- [Collect obproxy1,93cea56e873b82e6,a260758e8efe3667] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:22:40.031 INFO 4486 --- [Collect obproxy1,93cea56e873b82e6,a260758e8efe3667] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:22:40.031 INFO 4486 --- [Collect obproxy2,c1ceeebc75cf277b,3e99d10119ecef13] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:22:40.032 INFO 4486 --- [Collect obproxy2,c1ceeebc75cf277b,3e99d10119ecef13] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:22:40.050 INFO 4486 --- [Check obproxy2,902479d8c59521e7,3040ad19ae8307c2] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:22:40.050 INFO 4486 --- [Check obproxy2,902479d8c59521e7,3040ad19ae8307c2] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:22:40.052 INFO 4486 --- [Collect obproxy2,c1ceeebc75cf277b,3e99d10119ecef13] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:22:40.056 INFO 4486 --- [Check obproxy1,3ad9a5c603736c04,41b99cff95196ec5] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:22:40.056 INFO 4486 --- [Check obproxy1,3ad9a5c603736c04,41b99cff95196ec5] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:22:40.058 INFO 4486 --- [Collect obproxy2,c1ceeebc75cf277b,3e99d10119ecef13] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:22:40.063 INFO 4486 --- [Collect obproxy1,93cea56e873b82e6,a260758e8efe3667] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:22:40.064 INFO 4486 --- [Collect obproxy2,c1ceeebc75cf277b,3e99d10119ecef13] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:22:40.068 INFO 4486 --- [Collect obproxy1,93cea56e873b82e6,a260758e8efe3667] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:22:40.073 INFO 4486 --- [Collect obproxy1,93cea56e873b82e6,a260758e8efe3667] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:22:40.265 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e97c89e4cac13ea0,c9e206298d735f55] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e97c89e4cac13ea0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:40.265 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e97c89e4cac13ea0,c9e206298d735f55] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e97c89e4cac13ea0, duration=1 ms] 2024-10-16 13:22:40.889 INFO 4486 --- [ocp-async-4,0de1b15047adc29c,4a211c2c1e3f850f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:40.890 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b7586260612bc705,9cca6b9d74ebf1ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b7586260612bc705, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:40.890 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b7586260612bc705,9cca6b9d74ebf1ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b7586260612bc705, duration=1 ms] 2024-10-16 13:22:40.995 INFO 4486 --- [prometheus-hour-schedule-1,2bf80d682eccf269,6397014ad0980e39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:22:40.998 INFO 4486 --- [prometheus-minute-schedule-1,e10587eb45fc1e9b,4a040d23ba51740c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056180, interval=60 2024-10-16 13:22:41.004 INFO 4486 --- [prometheus-minute-schedule-1,e10587eb45fc1e9b,4a040d23ba51740c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:22:42.046 INFO 4486 --- [ocp-schedules-1,c4ccd3f5afaebe82,c21ec0b418347d4a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:22:42.425 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:22:42.889 INFO 4486 --- [ocp-async-1,ce94f16130c3d1dd,3f3bb5aedc42be19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:42.968 INFO 4486 --- [ocp-async-5,da7e883d5b272215,98fd468d952fc893] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:22:43.418 INFO 4486 --- [ocp-schedules-20,92824029dacd5ffa,02aa54f5ad727a29] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:22:43.420 INFO 4486 --- [ocp-schedules-20,92824029dacd5ffa,02aa54f5ad727a29] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:22:43.420 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:22:43.420 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:22:43.420 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:22:43.643 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9bdc4e5c60bb6439,8e27d8d9d308f39c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=9bdc4e5c60bb6439, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:22:43.644 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9bdc4e5c60bb6439,8e27d8d9d308f39c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=9bdc4e5c60bb6439, duration=2 ms] 2024-10-16 13:22:43.994 INFO 4486 --- [prometheus-minute-schedule-1,eb166abfe7f8c820,b2df7414fd071d5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056180, interval=60 2024-10-16 13:22:44.003 INFO 4486 --- [prometheus-minute-schedule-1,eb166abfe7f8c820,b2df7414fd071d5b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:22:44.889 INFO 4486 --- [ocp-async-7,b9990f549d076800,a159d24fd25cf783] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:45.278 INFO 4486 --- [ocp-schedules-16,d137affad73e443f,3405db8b94c50e97] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:22:45.278 INFO 4486 --- [ocp-schedules-16,db0c465c097c517a,4a651f4e591134cc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:22:45.280 INFO 4486 --- [ocp-schedules-16,db0c465c097c517a,4a651f4e591134cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:22:45.308 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.309 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:22:45.323 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.324 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:22:45.341 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.342 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:22:45.346 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.347 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.347 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:22:45.381 INFO 4486 --- [ocp-schedules-24,8f4bfaf6946c9f5c,55b31e891d7a5abc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:22:45.386 INFO 4486 --- [ocp-schedules-24,8f4bfaf6946c9f5c,55b31e891d7a5abc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:22:45.386 INFO 4486 --- [ocp-schedules-24,8f4bfaf6946c9f5c,55b31e891d7a5abc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:22:45.386 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.386 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.387 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:22:45.387 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:22:45.388 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.391 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:22:45.392 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:22:45.396 INFO 4486 --- [ocp-schedules-9,bb686d716fa385ad,1e432bba55380d8f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:22:45.396 INFO 4486 --- [ocp-schedules-9,bb686d716fa385ad,1e432bba55380d8f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:22:45.400 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.400 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:22:45.411 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.419 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.420 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.421 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:22:45.421 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:22:45.426 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:22:45.426 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:22:45.430 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:22:45.431 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:22:45.447 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.449 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.450 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:22:45.459 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:22:45.498 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:22:45.498 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:22:45.500 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.502 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.503 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:22:45.506 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.513 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:22:45.517 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:22:45.534 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:22:45.544 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.546 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.546 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:22:45.547 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.547 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:22:45.556 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:22:45.556 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:22:45.574 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.574 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:22:45.600 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.600 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:22:45.626 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.626 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:22:45.653 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.653 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:22:45.685 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:22:45.715 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.722 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.723 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:22:45.727 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:22:45.733 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:22:45.739 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:22:45.741 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:22:45.764 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:22:45.905 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:22:45.935 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:45.937 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:45.938 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:22:45.969 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:22:45.969 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055628392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:22:46.298 INFO 4486 --- [ocp-schedules-14,0279626eefcd7161,6461a8f0403823bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:22:46.323 WARN 4486 --- [scanner-13222189-1,0279626eefcd7161,f6699f595eae9e9e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:22:46.367 INFO 4486 --- [alarm-detect-11,0279626eefcd7161,0464ae702aead0a4] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056135 ~ 1729056156 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056150, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056150, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056150=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:22:46.599 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,030eb4fb80fddda8,4782313ec3b39847] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=030eb4fb80fddda8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:46.599 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,030eb4fb80fddda8,4782313ec3b39847] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=030eb4fb80fddda8, duration=1 ms] 2024-10-16 13:22:46.889 INFO 4486 --- [ocp-async-6,8de24a4f2d11ea97,0c988ccfbf2a9875] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:46.994 INFO 4486 --- [prometheus-minute-schedule-1,45b3c51df68de8f6,7d56d00523ca3cc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056180, interval=60 2024-10-16 13:22:47.002 INFO 4486 --- [prometheus-minute-schedule-1,45b3c51df68de8f6,7d56d00523ca3cc1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:22:47.013 INFO 4486 --- [ocp-schedules-7,93d70b7ac0a0f110,53ac6457cff28241] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:22:47.101 INFO 4486 --- [ocp-schedules-5,59e28e7aebf29146,7df5faaa1ea40b6d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:22:47.198 INFO 4486 --- [ocp-schedules-13,afa83272fb1d159e,736e5e276afb51a3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:22:48.426 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:22:48.806 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7830f903489a9d37,298f98703117a3e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7830f903489a9d37, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:48.806 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7830f903489a9d37,298f98703117a3e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7830f903489a9d37, duration=1 ms] 2024-10-16 13:22:48.889 INFO 4486 --- [ocp-async-1,e94d16d469005d4f,3bb2c5e5d29e0e07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:49.020 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7f67a5657aff1349,94850ecc13388087] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=7f67a5657aff1349, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:22:49.020 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7f67a5657aff1349,94850ecc13388087] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=7f67a5657aff1349, duration=1 ms] 2024-10-16 13:22:49.260 INFO 4486 --- [ocp-schedules-2,6898d9290d8340ba,3f26045795037aa3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:22:49.261 INFO 4486 --- [ocp-schedules-2,6898d9290d8340ba,3f26045795037aa3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:22:49.261 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:22:49.261 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:22:49.261 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:22:49.261 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:22:49.261 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:22:49.267 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:22:49.267 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:22:49.267 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:22:49.267 INFO 4486 --- [ocp-schedules-2,6898d9290d8340ba,3f26045795037aa3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:22:49.267 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:22:49.268 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:22:49.268 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:22:49.268 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:22:49.269 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:22:49.269 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:22:49.269 INFO 4486 --- [ocp-schedules-2,6898d9290d8340ba,3f26045795037aa3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:22:49.269 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:22:49.290 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:22:49.290 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:22:49.292 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:22:49.293 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:22:49.294 INFO 4486 --- [ocp-schedules-2,6898d9290d8340ba,3f26045795037aa3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:22:49.295 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:22:49.315 INFO 4486 --- [ocp-schedules-2,6898d9290d8340ba,3f26045795037aa3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:22:49.926 INFO 4486 --- [ocp-schedules-5,d75d4b9c0fd05ff9,c9808b652abab01a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:22:49.994 INFO 4486 --- [prometheus-minute-schedule-1,5237c6581d2023c5,b3d1da695d5311b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056180, interval=60 2024-10-16 13:22:50.000 INFO 4486 --- [prometheus-minute-schedule-1,5237c6581d2023c5,b3d1da695d5311b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:22:50.270 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b01c59cdf946145b,71bdb456626d3d60] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b01c59cdf946145b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:50.270 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b01c59cdf946145b,71bdb456626d3d60] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b01c59cdf946145b, duration=2 ms] 2024-10-16 13:22:50.297 INFO 4486 --- [ocp-schedules-19,da37561dcd9652b1,aae2e3af4b910804] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:22:50.297 INFO 4486 --- [ocp-schedules-19,da37561dcd9652b1,aae2e3af4b910804] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:22:50.298 INFO 4486 --- [ocp-schedules-19,da37561dcd9652b1,aae2e3af4b910804] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:22:50.299 INFO 4486 --- [ocp-schedules-19,da37561dcd9652b1,aae2e3af4b910804] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:22:50.299 INFO 4486 --- [ocp-schedules-19,da37561dcd9652b1,aae2e3af4b910804] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:22:50.299 INFO 4486 --- [ocp-schedules-19,da37561dcd9652b1,aae2e3af4b910804] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703033600770641920 2024-10-16 13:22:50.890 INFO 4486 --- [ocp-async-7,5380bbe055aa7181,f02580acaaf8ef77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:50.894 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8f8359d6c867eacb,06e9b3d4862ffa50] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8f8359d6c867eacb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:50.895 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8f8359d6c867eacb,06e9b3d4862ffa50] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8f8359d6c867eacb, duration=2 ms] 2024-10-16 13:22:51.427 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:22:52.949 INFO 4486 --- [ocp-async-4,c8cb9f500a849864,e12914f5e06eb393] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:52.969 INFO 4486 --- [ocp-async-10,99fd0b5be416421f,1038ad74d87c984e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:22:52.994 INFO 4486 --- [prometheus-minute-schedule-1,dfa8613d4c975c00,9cba502a1b773609] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056180, interval=60 2024-10-16 13:22:53.002 INFO 4486 --- [prometheus-minute-schedule-1,dfa8613d4c975c00,9cba502a1b773609] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:22:53.421 INFO 4486 --- [ocp-schedules-20,92824029dacd5ffa,02aa54f5ad727a29] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:22:53.857 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:22:53.858 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:22:53.864 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:22:53.864 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:22:53.864 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:22:53.865 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:22:53.870 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:22:53.870 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:22:53.870 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:22:53.870 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:22:53.918 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:22:53.925 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:22:53.925 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:22:53.925 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:22:53.925 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:22:53.925 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:22:53.932 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:22:54.308 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:22:54.428 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:22:54.731 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.861887561S 2024-10-16 13:22:54.732 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:22:54.797 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:22:54.797 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:22:54.823 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:22:54.893 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:22:54.893 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:22:54.893 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:22:54.893 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:22:54.893 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:22:54.894 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:22:54.902 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:22:54.949 INFO 4486 --- [ocp-async-2,61fd943a6ccd75a1,72463fb862cf533a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:55.071 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] o.s.boot.SpringApplication : Started application in 1.206 seconds (JVM running for 77078.122) 2024-10-16 13:22:55.071 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.20671736S 2024-10-16 13:22:55.406 INFO 4486 --- [ocp-schedules-27,25873613bf7b4257,a33310b11bcff1fe] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:22:55.994 INFO 4486 --- [prometheus-minute-schedule-1,e8197ab2aebe573d,82421e3a9accef8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056180, interval=60 2024-10-16 13:22:56.000 INFO 4486 --- [prometheus-minute-schedule-1,e8197ab2aebe573d,82421e3a9accef8f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:22:56.423 INFO 4486 --- [ocp-schedules-4,88002d1321f4d624,350ccea7bb333159] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:22:56.455 WARN 4486 --- [scanner-13222403-1,88002d1321f4d624,4db9012fe246356a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:22:56.474 INFO 4486 --- [alarm-detect-19,88002d1321f4d624,a3d92bfacbb19922] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:22:56.478 INFO 4486 --- [alarm-detect-19,88002d1321f4d624,a3d92bfacbb19922] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:22:56.482 INFO 4486 --- [alarm-detect-19,88002d1321f4d624,a3d92bfacbb19922] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:22:56.493 INFO 4486 --- [alarm-detect-19,88002d1321f4d624,3765ad62716bc80b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:22:56.494 INFO 4486 --- [alarm-detect-3,88002d1321f4d624,88580165263a0652] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:22:56.498 INFO 4486 --- [alarm-detect-3,88002d1321f4d624,88580165263a0652] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:22:56.498 INFO 4486 --- [alarm-detect-19,88002d1321f4d624,3765ad62716bc80b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:22:56.501 INFO 4486 --- [alarm-detect-19,88002d1321f4d624,3765ad62716bc80b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:22:56.503 INFO 4486 --- [alarm-detect-3,88002d1321f4d624,88580165263a0652] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:22:56.512 INFO 4486 --- [alarm-detect-1,88002d1321f4d624,de2fead062d5d299] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056150 ~ 1729056166 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056165, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056165, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056165=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:22:56.561 INFO 4486 --- [alarm-detect-3,88002d1321f4d624,4b7ed3b064d91635] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:22:56.566 INFO 4486 --- [alarm-detect-3,88002d1321f4d624,4b7ed3b064d91635] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:22:56.570 INFO 4486 --- [alarm-detect-3,88002d1321f4d624,4b7ed3b064d91635] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:22:56.605 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,cd105b9672d2103a,dd07bb6be82c4bc5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cd105b9672d2103a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:56.605 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,cd105b9672d2103a,dd07bb6be82c4bc5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cd105b9672d2103a, duration=2 ms] 2024-10-16 13:22:56.949 INFO 4486 --- [ocp-async-9,c69bae8bec48cc82,4fe87a0210503de4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:57.429 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:22:58.090 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:22:58.107 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:22:58.109 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:22:58.141 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:22:58.143 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:22:58.168 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:22:58.170 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:22:58.200 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:22:58.202 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:22:58.228 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:22:58.229 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:22:58.250 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:22:58.251 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:22:58.280 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:22:58.281 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:22:58.309 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:22:58.310 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:22:58.335 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:22:58.336 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:22:58.363 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:22:58.364 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:22:58.388 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:22:58.390 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:22:58.409 INFO 4486 --- [ocp-schedules-24,b04af4d7761e2683,4a115f1afecffcc0] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:22:58.707 INFO 4486 --- [ocp-schedules-2,3665fff1318b2df4,99151f7763aa86e1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:22:58.811 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,98f09480d79bf223,ce7b4032b46deb2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=98f09480d79bf223, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:22:58.811 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,98f09480d79bf223,ce7b4032b46deb2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=98f09480d79bf223, duration=2 ms] 2024-10-16 13:22:58.950 INFO 4486 --- [ocp-async-3,940fd9517c1fe305,e17a48bc212942a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:22:58.994 INFO 4486 --- [prometheus-minute-schedule-1,a074cf7c596424a4,839ca7990a4cb695] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056180, interval=60 2024-10-16 13:22:59.001 INFO 4486 --- [prometheus-minute-schedule-1,a074cf7c596424a4,839ca7990a4cb695] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:22:59.283 INFO 4486 --- [ocp-schedules-10,cdc86c3df6a4d048,172834d53d831852] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-10-16 13:22:59.286 INFO 4486 --- [ocp-schedules-10,cdc86c3df6a4d048,172834d53d831852] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:22:59.315 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:59.317 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:59.319 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:03:03.102283, 2024-10-16 13:22:59.292, 1000] 2024-10-16 13:22:59.321 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-10-16 02:03:03.102283, endTime 2024-10-16 13:22:59.292 2024-10-16 13:22:59.322 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:42:50.879398, 2024-10-16 13:22:59.292, 1000] 2024-10-16 13:22:59.328 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 5, startTime 2024-10-16 08:42:50.879398, endTime 2024-10-16 13:22:59.292 2024-10-16 13:22:59.332 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:59.336 INFO 4486 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:59.336 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:06:51.05999, 2024-10-16 13:22:59.292, 1000] 2024-10-16 13:22:59.340 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:59.343 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000003, startTime 2024-10-16 02:06:51.05999, endTime 2024-10-16 13:22:59.292 2024-10-16 13:22:59.348 INFO 4486 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:59.350 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:59.376 INFO 4486 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:22:59.378 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:22:59.380 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 12:59:45.930954, 2024-10-16 13:22:59.332, 1000] 2024-10-16 13:22:59.384 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 5, startTime 2024-10-16 12:59:45.930954, endTime 2024-10-16 13:22:59.332 2024-10-16 13:22:59.396 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:00:14.755425, 2024-10-16 13:22:59.326, 1000] 2024-10-16 13:22:59.397 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-10-16 08:00:14.755425, endTime 2024-10-16 13:22:59.326 2024-10-16 13:22:59.405 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-12 15:26:31.336743, 2024-10-16 13:22:59.346, 1000] 2024-10-16 13:22:59.410 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000003, startTime 2024-10-12 15:26:31.336743, endTime 2024-10-16 13:22:59.346 2024-10-16 13:22:59.931 INFO 4486 --- [ocp-schedules-17,922413c95d82c31e,36d6bd096bd3af5f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:23:00.073 INFO 4486 --- [ocp-schedules-19,7df67c2df280cc6b,793984c827cf7908] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:23:00.073+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:23:00.076 INFO 4486 --- [ocp-schedules-19,7df67c2df280cc6b,793984c827cf7908] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:23:00.084 INFO 4486 --- [ocp-schedules-19,7df67c2df280cc6b,793984c827cf7908] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:23:00.094 INFO 4486 --- [ocp-schedules-19,7df67c2df280cc6b,793984c827cf7908] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:23:00.276 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f4cecce0385a38c2,a18e8b012bb54256] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f4cecce0385a38c2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:00.277 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f4cecce0385a38c2,a18e8b012bb54256] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f4cecce0385a38c2, duration=2 ms] 2024-10-16 13:23:00.430 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:00.899 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,27f4f85ecc0a1371,205d6ebb90e3640d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=27f4f85ecc0a1371, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:00.899 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,27f4f85ecc0a1371,205d6ebb90e3640d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=27f4f85ecc0a1371, duration=1 ms] 2024-10-16 13:23:00.950 INFO 4486 --- [ocp-async-1,35c48277c2dc5b03,367d2257a6447f9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:01.994 INFO 4486 --- [prometheus-minute-schedule-1,efbc7fffac4b8bb5,c9a9c3076d2034aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:01.999 INFO 4486 --- [prometheus-minute-schedule-1,efbc7fffac4b8bb5,c9a9c3076d2034aa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:02.950 INFO 4486 --- [ocp-async-2,b5df444b923434c0,a0befce0c5c7cfbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:02.969 INFO 4486 --- [ocp-async-9,35d663de67324b12,b2978bcb6743dde4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:23:03.431 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:04.950 INFO 4486 --- [ocp-async-6,337f9a64c17e6b6b,c06647313eb8eec3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:04.994 INFO 4486 --- [prometheus-minute-schedule-1,826751e49b489251,d891bae36540abb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:05.000 INFO 4486 --- [prometheus-minute-schedule-1,826751e49b489251,d891bae36540abb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:06.432 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:06.591 INFO 4486 --- [ocp-schedules-29,d925f108de8f67e7,ef3f698259933f00] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:23:06.621 WARN 4486 --- [scanner-13222740-1,d925f108de8f67e7,19a65f4753c2b57b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:23:06.623 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,22600d69943a85bc,07cd78dff8c24d68] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=22600d69943a85bc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:06.623 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,22600d69943a85bc,07cd78dff8c24d68] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=22600d69943a85bc, duration=6 ms] 2024-10-16 13:23:06.950 INFO 4486 --- [ocp-async-4,0a758010a6e7d575,6644fd6853015c66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:06.969 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eae1be81f29734f1,e0f6580015fd5205] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=eae1be81f29734f1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:23:06.969 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eae1be81f29734f1,e0f6580015fd5205] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=eae1be81f29734f1, duration=1 ms] 2024-10-16 13:23:07.066 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:23:07.067 INFO 4486 --- [ocp-schedules-8,1dcee6f7ace38d45,eee3a4343ca89e2f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:23:07.069 INFO 4486 --- [ocp-schedules-8,1dcee6f7ace38d45,eee3a4343ca89e2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:23:07.081 INFO 4486 --- [ocp-schedules-6,1a3214d73b7657a1,abcf90f6ed7a3a0e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:23:07.083 INFO 4486 --- [ocp-schedules-6,1a3214d73b7657a1,abcf90f6ed7a3a0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:23:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:23:07.089 INFO 4486 --- [ocp-schedules-25,f3617b833209313f,cef4e972fc579013] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:23:07.090 INFO 4486 --- [ocp-schedules-13,d05b490e32dc4ab0,cad2158f2791b038] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:23:07.090 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:23:07.090 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:23:07.090 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:23:07.091 INFO 4486 --- [ocp-schedules-5,7024d3b0771e580c,9e8b8e5a11b64195] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:23:07.091 INFO 4486 --- [host-check-agent-status1,d05b490e32dc4ab0,e3190e3ac8cf7350] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:23:07.091 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:23:07.091 INFO 4486 --- [host-check-agent-status2,d05b490e32dc4ab0,ecf32c2a0a808e45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:23:07.092 INFO 4486 --- [host-check-clockdiff1,7024d3b0771e580c,4cfffecffbbd8b94] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:23:07.092 INFO 4486 --- [host-check-agent-status4,d05b490e32dc4ab0,babe309010596753] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:23:07.093 INFO 4486 --- [host-check-agent-status3,d05b490e32dc4ab0,dd9e276889fc52d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:23:07.093 INFO 4486 --- [host-check-clockdiff3,7024d3b0771e580c,74b124fab71de0c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:23:07.093 INFO 4486 --- [host-check-clockdiff2,7024d3b0771e580c,710db6ee44c7ff0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:23:07.093 INFO 4486 --- [host-check-clockdiff4,7024d3b0771e580c,593215bc01c0ce01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:23:07.098 INFO 4486 --- [host-check-agent-status4,d05b490e32dc4ab0,babe309010596753] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:23:07.100 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:07.101 INFO 4486 --- [host-check-agent-status3,d05b490e32dc4ab0,dd9e276889fc52d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:23:07.101 INFO 4486 --- [host-check-agent-status1,d05b490e32dc4ab0,e3190e3ac8cf7350] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:07.101 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:23:07.124 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:23:07.124 INFO 4486 --- [host-check-agent-status4,d05b490e32dc4ab0,babe309010596753] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:23:07.135 INFO 4486 --- [host-check-clockdiff4,7024d3b0771e580c,593215bc01c0ce01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056187 2 2 , error= 2024-10-16 13:23:07.135 INFO 4486 --- [host-check-clockdiff2,7024d3b0771e580c,710db6ee44c7ff0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056187 3 3 , error= 2024-10-16 13:23:07.135 INFO 4486 --- [host-check-agent-status3,d05b490e32dc4ab0,dd9e276889fc52d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:23:07.140 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:07.140 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:23:07.143 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:23:07.143 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:23:07.146 INFO 4486 --- [host-check-agent-status1,d05b490e32dc4ab0,e3190e3ac8cf7350] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:07.146 INFO 4486 --- [host-check-clockdiff1,7024d3b0771e580c,4cfffecffbbd8b94] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056187 0 0 , error= 2024-10-16 13:23:07.146 INFO 4486 --- [host-check-clockdiff3,7024d3b0771e580c,74b124fab71de0c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056187 3 3 , error= 2024-10-16 13:23:07.146 INFO 4486 --- [host-check-agent-status2,d05b490e32dc4ab0,ecf32c2a0a808e45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:23:07.148 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:23:07.148 INFO 4486 --- [host-check-agent-status2,d05b490e32dc4ab0,ecf32c2a0a808e45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:23:07.157 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.158 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:07.159 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:23:07.173 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:23:07.174 INFO 4486 --- [host-check-agent-status3,d05b490e32dc4ab0,dd9e276889fc52d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:23:07.176 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:23:07.176 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:23:07.175730095+08:00 2024-10-16 13:23:07.176 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:23:07.176 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:23:07.176 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:23:07.176+08:00 2024-10-16 13:23:07.176 INFO 4486 --- [host-check-agent-status1,d05b490e32dc4ab0,e3190e3ac8cf7350] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:23:07.177 INFO 4486 --- [host-check-agent-status4,d05b490e32dc4ab0,babe309010596753] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:23:07.178 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:23:07.180544273+08:00 2024-10-16 13:23:07.178 INFO 4486 --- [host-check-agent-status2,d05b490e32dc4ab0,ecf32c2a0a808e45] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:23:07.179 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:23:07.178+08:00 2024-10-16 13:23:07.179 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:23:07.182047130+08:00 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status1,d05b490e32dc4ab0,e3190e3ac8cf7350] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status3,d05b490e32dc4ab0,dd9e276889fc52d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status1,d05b490e32dc4ab0,e3190e3ac8cf7350] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status1,d05b490e32dc4ab0,e3190e3ac8cf7350] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status3,d05b490e32dc4ab0,dd9e276889fc52d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status3,d05b490e32dc4ab0,dd9e276889fc52d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status4,d05b490e32dc4ab0,babe309010596753] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status4,d05b490e32dc4ab0,babe309010596753] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:23:07.183 INFO 4486 --- [host-check-agent-status4,d05b490e32dc4ab0,babe309010596753] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:23:07.184 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:23:07.179+08:00 2024-10-16 13:23:07.184 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:23:07.182297520+08:00 2024-10-16 13:23:07.184 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:23:07.184+08:00 2024-10-16 13:23:07.184 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.185 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:07.190 INFO 4486 --- [host-check-agent-status2,d05b490e32dc4ab0,ecf32c2a0a808e45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:23:07.190 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:23:07.190 INFO 4486 --- [host-check-agent-status2,d05b490e32dc4ab0,ecf32c2a0a808e45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:23:07.190 INFO 4486 --- [host-check-agent-status2,d05b490e32dc4ab0,ecf32c2a0a808e45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:23:07.195 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:23:07.176+08:00 and status AVAILABLE 2024-10-16 13:23:07.196 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:23:07.179+08:00 and status AVAILABLE 2024-10-16 13:23:07.197 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:23:07.198 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:23:07.199 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:23:07.184+08:00 and status AVAILABLE 2024-10-16 13:23:07.199 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.202 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:23:07.178+08:00 and status AVAILABLE 2024-10-16 13:23:07.202 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:23:07.204 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:23:07.204 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:23:07.206 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:07.207 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:23:07.208 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:07.208 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:23:07.211 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:23:07.213 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:23:07.213 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:23:07.214 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:23:07.216 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.217 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:23:07.227 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.228 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:23:07.229 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:07.234 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:07.234 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:07.236 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:07.243 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.244 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:23:07.246 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.248 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:23:07.259 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:07.260 INFO 4486 --- [ocp-schedules-4,70b9f6302bde8ec0,06ddb939af1a2650] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:23:07.264 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:23:07.266 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:23:07.273 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:23:07.277 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:23:07.280 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:23:07.286 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:23:07.289 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:23:07.300 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:23:07.305 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:07.309 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:07.310 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:07.315 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:07.335 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:23:07.347 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:23:07.350 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:23:07.355 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:23:07.365 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:07.365 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:23:07.367 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:23:07.368 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:23:07.380 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:23:07.381 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:07.382 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:07.394 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:23:07.397 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:07.405 INFO 4486 --- [host-check-host-status3,f3617b833209313f,20a68a6b602a546b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:23:07.416 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:23:07.420 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:23:07.426 INFO 4486 --- [host-check-host-status4,f3617b833209313f,ab460aac762e0bbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:23:07.430 INFO 4486 --- [host-check-host-status2,f3617b833209313f,8af5a209794a6f0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:23:07.454 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:23:07.466 INFO 4486 --- [host-check-host-status1,f3617b833209313f,de350ec33a83cf04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:23:07.994 INFO 4486 --- [prometheus-minute-schedule-1,c34c9d8ed303b8d9,5ea68f2c134be807] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:08.000 INFO 4486 --- [prometheus-minute-schedule-1,c34c9d8ed303b8d9,5ea68f2c134be807] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:08.823 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2a3cfe7869209a6b,89d649821ac151d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2a3cfe7869209a6b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:08.823 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2a3cfe7869209a6b,89d649821ac151d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2a3cfe7869209a6b, duration=6 ms] 2024-10-16 13:23:08.951 INFO 4486 --- [ocp-async-5,6e5a8a5527e15ba2,ff94edecc5d538cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:09.411 INFO 4486 --- [ocp-schedules-10,cdc86c3df6a4d048,172834d53d831852] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=3, successCount=3 2024-10-16 13:23:09.434 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:09.936 INFO 4486 --- [ocp-schedules-2,b1b4824b39b3d946,0ee6ab611971c501] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:23:10.008 INFO 4486 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:23:10.009 INFO 4486 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:23:10.030 INFO 4486 --- [Check obproxy1,d64255cbf07fc3d7,44086d57f9852ef6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:23:10.030 INFO 4486 --- [Check obproxy2,3ee3d697bf554c93,c94b0f686d8233a4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:23:10.035 INFO 4486 --- [Check obproxy1,d64255cbf07fc3d7,44086d57f9852ef6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:10.036 INFO 4486 --- [Collect obproxy2,49e7029c0708cfda,842d1dd2ebee4846] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:23:10.036 INFO 4486 --- [Collect obproxy2,49e7029c0708cfda,842d1dd2ebee4846] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:10.037 INFO 4486 --- [Collect obproxy1,63f78de571f779f1,a853e024fd5f7d22] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:23:10.037 INFO 4486 --- [Collect obproxy1,63f78de571f779f1,a853e024fd5f7d22] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:10.037 INFO 4486 --- [Check obproxy2,3ee3d697bf554c93,c94b0f686d8233a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:10.063 INFO 4486 --- [Collect obproxy2,49e7029c0708cfda,842d1dd2ebee4846] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:23:10.063 INFO 4486 --- [Check obproxy2,3ee3d697bf554c93,c94b0f686d8233a4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:23:10.063 INFO 4486 --- [Check obproxy2,3ee3d697bf554c93,c94b0f686d8233a4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:23:10.066 INFO 4486 --- [Check obproxy1,d64255cbf07fc3d7,44086d57f9852ef6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:23:10.066 INFO 4486 --- [Check obproxy1,d64255cbf07fc3d7,44086d57f9852ef6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:23:10.067 INFO 4486 --- [Collect obproxy2,49e7029c0708cfda,842d1dd2ebee4846] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:23:10.068 INFO 4486 --- [Collect obproxy1,63f78de571f779f1,a853e024fd5f7d22] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:23:10.071 INFO 4486 --- [Collect obproxy1,63f78de571f779f1,a853e024fd5f7d22] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:23:10.072 INFO 4486 --- [Collect obproxy2,49e7029c0708cfda,842d1dd2ebee4846] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:23:10.073 INFO 4486 --- [Collect obproxy1,63f78de571f779f1,a853e024fd5f7d22] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:23:10.282 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,61b3606c08b6d91e,b8de5c7185e4bada] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=61b3606c08b6d91e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:10.282 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,61b3606c08b6d91e,b8de5c7185e4bada] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=61b3606c08b6d91e, duration=1 ms] 2024-10-16 13:23:10.903 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3e7b3734fd3f2266,c71d43afb5946010] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3e7b3734fd3f2266, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:10.903 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3e7b3734fd3f2266,c71d43afb5946010] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3e7b3734fd3f2266, duration=1 ms] 2024-10-16 13:23:10.951 INFO 4486 --- [ocp-async-8,145b2e837586cb30,2757ff3f02023941] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:10.980 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:23:10.994 INFO 4486 --- [prometheus-minute-schedule-1,7cbd050f07cb73cc,6a41e4d511d8b3de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:10.999 INFO 4486 --- [prometheus-minute-schedule-1,7cbd050f07cb73cc,6a41e4d511d8b3de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:23:12.435 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:12.757 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056120, nodeCount=450, affectRows=450 2024-10-16 13:23:12.758 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056120, nodeCount=390, affectRows=390 2024-10-16 13:23:12.760 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056120, nodeCount=450, validCount=450 2024-10-16 13:23:12.760 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056120, nodeCount=390, validCount=390 2024-10-16 13:23:12.762 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056120, nodeCount=475, affectRows=475 2024-10-16 13:23:12.763 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056120, nodeCount=544, affectRows=544 2024-10-16 13:23:12.764 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056120, nodeCount=509, affectRows=509 2024-10-16 13:23:12.764 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056120, nodeCount=474, validCount=474 2024-10-16 13:23:12.766 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056120, nodeCount=582, affectRows=582 2024-10-16 13:23:12.766 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056120, nodeCount=543, validCount=543 2024-10-16 13:23:12.766 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056120, nodeCount=508, validCount=508 2024-10-16 13:23:12.768 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056120, nodeCount=582, validCount=582 2024-10-16 13:23:12.906 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056120, nodeCount=500, affectRows=500 2024-10-16 13:23:12.908 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056120, nodeCount=500, validCount=500 2024-10-16 13:23:12.951 INFO 4486 --- [ocp-async-10,95c104262bb9b34d,47ed8d75bb537b91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:12.969 INFO 4486 --- [ocp-async-4,9e266ddbcdac06b0,bf1d78d90234a30a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:23:13.994 INFO 4486 --- [prometheus-minute-schedule-1,761153fb276e1a09,c6d7a9aefda1e78a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:14.001 INFO 4486 --- [prometheus-minute-schedule-1,761153fb276e1a09,c6d7a9aefda1e78a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:14.215 INFO 4486 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056120, nodeCount=480, affectRows=480 2024-10-16 13:23:14.911 INFO 4486 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056120, nodeCount=480, validCount=480 2024-10-16 13:23:14.951 INFO 4486 --- [ocp-async-2,4b25d384d83c598f,e6a2f0ec03cfa3f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:15.435 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:15.970 INFO 4486 --- [ocp-schedules-16,db0c465c097c517a,4a651f4e591134cc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:23:15.970 INFO 4486 --- [ocp-schedules-16,57ae5d81135ab87c,a1b8cfa5a95abfc6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:23:15.973 INFO 4486 --- [ocp-schedules-16,57ae5d81135ab87c,a1b8cfa5a95abfc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:23:16.007 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.008 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:16.009 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.010 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:16.040 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.040 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.041 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:16.044 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.044 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:23:16.075 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.075 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:16.077 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:23:16.086 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.089 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:23:16.091 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:16.109 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.109 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:16.128 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.138 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:16.140 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:23:16.145 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:23:16.150 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:16.152 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:16.163 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.164 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.165 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:23:16.176 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:23:16.184 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:23:16.184 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:23:16.213 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.215 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.215 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:16.219 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.222 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:23:16.226 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:16.248 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:23:16.252 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.252 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:16.259 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.264 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.265 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:23:16.274 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:23:16.274 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:23:16.278 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.278 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:16.302 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.302 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:16.329 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.329 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:16.353 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.353 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:16.380 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:16.407 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.409 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.410 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:16.414 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:23:16.419 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:23:16.424 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:23:16.426 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:23:16.448 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:23:16.580 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:23:16.609 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:16.610 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:16.611 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:23:16.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d57d73a479cb4567,f671ecf8d1685785] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d57d73a479cb4567, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:16.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d57d73a479cb4567,f671ecf8d1685785] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d57d73a479cb4567, duration=2 ms] 2024-10-16 13:23:16.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:23:16.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055628392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:23:16.797 INFO 4486 --- [ocp-schedules-29,00dcf231249217bd,97a6f2cff0c0fd55] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:23:16.841 WARN 4486 --- [scanner-13223815-1,00dcf231249217bd,04b6c481fd626ae9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:23:16.856 INFO 4486 --- [alarm-detect-2,00dcf231249217bd,73edb298101fce6a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:16.861 INFO 4486 --- [alarm-detect-2,00dcf231249217bd,73edb298101fce6a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:16.866 INFO 4486 --- [alarm-detect-2,00dcf231249217bd,73edb298101fce6a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:16.894 INFO 4486 --- [alarm-detect-2,00dcf231249217bd,fca1a58f72edc8b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:16.898 INFO 4486 --- [alarm-detect-14,00dcf231249217bd,19523b851867eaa9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:16.901 INFO 4486 --- [alarm-detect-2,00dcf231249217bd,fca1a58f72edc8b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:16.906 INFO 4486 --- [alarm-detect-14,00dcf231249217bd,19523b851867eaa9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:16.911 INFO 4486 --- [alarm-detect-2,00dcf231249217bd,fca1a58f72edc8b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:16.915 INFO 4486 --- [alarm-detect-14,00dcf231249217bd,19523b851867eaa9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:16.930 INFO 4486 --- [alarm-detect-17,00dcf231249217bd,61ebe3344076adfb] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056165 ~ 1729056186 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056180, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056180, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056180=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:23:16.952 INFO 4486 --- [ocp-async-3,842f65b0b897968e,3cdc4b5acfbbae51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:16.979 INFO 4486 --- [alarm-detect-5,00dcf231249217bd,bad4f1e1f748ad5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:16.983 INFO 4486 --- [alarm-detect-5,00dcf231249217bd,bad4f1e1f748ad5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:16.988 INFO 4486 --- [alarm-detect-5,00dcf231249217bd,bad4f1e1f748ad5e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:16.995 INFO 4486 --- [prometheus-minute-schedule-1,a24cff992d6ceb58,2baaaafef56146cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:17.002 INFO 4486 --- [prometheus-minute-schedule-1,a24cff992d6ceb58,2baaaafef56146cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:17.016 INFO 4486 --- [ocp-schedules-24,c0ff92962f7c69a6,003a0cbd938d8895] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:23:17.071 INFO 4486 --- [ocp-schedules-8,1dcee6f7ace38d45,eee3a4343ca89e2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:23:17.190 INFO 4486 --- [ocp-schedules-6,1a3214d73b7657a1,abcf90f6ed7a3a0e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:23:18.437 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:18.712 INFO 4486 --- [ocp-schedules-30,c2e1c6930e5cbc88,8bbfb16dc56a0d3b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:23:18.827 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6f194a443c6d650a,1c785395d451fd85] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6f194a443c6d650a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:18.828 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6f194a443c6d650a,1c785395d451fd85] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6f194a443c6d650a, duration=1 ms] 2024-10-16 13:23:18.952 INFO 4486 --- [ocp-async-10,2229b4632c109cf5,fdb56eebefcad12f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:19.942 INFO 4486 --- [ocp-schedules-20,acd97c512d67dc43,7e2648a829586d78] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:23:19.994 INFO 4486 --- [prometheus-minute-schedule-1,8f033f16c05a3cb7,f7dd2be44c23743e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:20.001 INFO 4486 --- [prometheus-minute-schedule-1,8f033f16c05a3cb7,f7dd2be44c23743e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:20.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,09dba23c4fd6d955,1c3bca1617a7715b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=09dba23c4fd6d955, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:20.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,09dba23c4fd6d955,1c3bca1617a7715b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=09dba23c4fd6d955, duration=1 ms] 2024-10-16 13:23:20.907 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5bb6c344d2ba236d,cd933f754ff21d52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5bb6c344d2ba236d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:20.907 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5bb6c344d2ba236d,cd933f754ff21d52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5bb6c344d2ba236d, duration=1 ms] 2024-10-16 13:23:20.952 INFO 4486 --- [ocp-async-4,008910d3f8aca10b,b8970aa00c8aa589] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:20.996 INFO 4486 --- [prometheus-cache-monitor1,924fafb025f3ebbd,d88afeecff30afd4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8244} 2024-10-16 13:23:20.996 INFO 4486 --- [prometheus-cache-monitor1,924fafb025f3ebbd,d88afeecff30afd4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:23:21.438 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:22.362 INFO 4486 --- [ocp-schedules-3,09b900204371bc38,fad9f54142602e7f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:23:22.384 INFO 4486 --- [ocp-schedules-22,f046f9bf20b543df,aa6bcf287eeb29c9] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:23:22.952 INFO 4486 --- [ocp-async-5,cddafc8c244b7671,0750ca421bc10833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:22.969 INFO 4486 --- [ocp-async-3,40cfc0f0624f1834,982e4398f5579388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:23:22.994 INFO 4486 --- [prometheus-minute-schedule-1,f6d1a63410fa7baa,f89683d78291d0ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:22.998 INFO 4486 --- [prometheus-minute-schedule-1,f6d1a63410fa7baa,f89683d78291d0ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:23.688 INFO 4486 --- [ocp-schedules-22,1f917e5f17e59abb,c8819731fe39b0a6] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:23:23.688 INFO 4486 --- [ocp-schedules-22,1f917e5f17e59abb,c8819731fe39b0a6] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:23:23.722 INFO 4486 --- [Collect all tenants major compactions1,1f917e5f17e59abb,c4f55cddbeb52ab6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:23:23.723 INFO 4486 --- [Collect all tenants major compactions2,1f917e5f17e59abb,eaea5fce1b340234] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:23:23.723 INFO 4486 --- [Collect all tenants major compactions4,1f917e5f17e59abb,3b35516f8d725b40] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:23:23.723 INFO 4486 --- [Collect all tenants major compactions5,1f917e5f17e59abb,df998845064a1d45] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:23:23.724 INFO 4486 --- [Collect all tenants major compactions3,1f917e5f17e59abb,d7542b31fa07334f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:23:23.738 INFO 4486 --- [Collect all tenants major compactions1,1f917e5f17e59abb,c4f55cddbeb52ab6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:23.739 INFO 4486 --- [Collect all tenants major compactions1,1f917e5f17e59abb,c4f55cddbeb52ab6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:23:23.739 INFO 4486 --- [Collect all tenants major compactions3,1f917e5f17e59abb,d7542b31fa07334f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:23.740 INFO 4486 --- [Collect all tenants major compactions2,1f917e5f17e59abb,eaea5fce1b340234] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:23.741 INFO 4486 --- [Collect all tenants major compactions3,1f917e5f17e59abb,d7542b31fa07334f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:23:23.741 INFO 4486 --- [Collect all tenants major compactions2,1f917e5f17e59abb,eaea5fce1b340234] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:23:23.742 INFO 4486 --- [Collect all tenants major compactions1,1f917e5f17e59abb,c4f55cddbeb52ab6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:23:23.744 INFO 4486 --- [Collect all tenants major compactions2,1f917e5f17e59abb,eaea5fce1b340234] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:23:23.744 INFO 4486 --- [Collect all tenants major compactions3,1f917e5f17e59abb,d7542b31fa07334f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:23:23.746 INFO 4486 --- [Collect all tenants major compactions1,1f917e5f17e59abb,c4f55cddbeb52ab6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:23:23.749 INFO 4486 --- [Collect all tenants major compactions2,1f917e5f17e59abb,eaea5fce1b340234] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:23:23.750 INFO 4486 --- [Collect all tenants major compactions3,1f917e5f17e59abb,d7542b31fa07334f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:23:23.753 INFO 4486 --- [Collect all tenants major compactions5,1f917e5f17e59abb,df998845064a1d45] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:23.754 INFO 4486 --- [Collect all tenants major compactions4,1f917e5f17e59abb,3b35516f8d725b40] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:23.755 INFO 4486 --- [Collect all tenants major compactions5,1f917e5f17e59abb,df998845064a1d45] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:23:23.756 INFO 4486 --- [Collect all tenants major compactions4,1f917e5f17e59abb,3b35516f8d725b40] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:23:23.759 INFO 4486 --- [Collect all tenants major compactions4,1f917e5f17e59abb,3b35516f8d725b40] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:23:23.759 INFO 4486 --- [Collect all tenants major compactions4,1f917e5f17e59abb,3b35516f8d725b40] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:23:23.760 INFO 4486 --- [Collect all tenants major compactions5,1f917e5f17e59abb,df998845064a1d45] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:23:23.760 INFO 4486 --- [Collect all tenants major compactions5,1f917e5f17e59abb,df998845064a1d45] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:23:23.761 INFO 4486 --- [ocp-schedules-22,1f917e5f17e59abb,c8819731fe39b0a6] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:23:23.816 INFO 4486 --- [ocp-schedules-11,9fef686bffabeb18,c9a4630979756f72] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:23:24.438 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:24.952 INFO 4486 --- [ocp-async-6,b1ccd8a5395e8bd4,3f7dbe71b927fd48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:25.994 INFO 4486 --- [prometheus-minute-schedule-1,2faf2bb0aa31f0a0,3e94ab6c3aa57aa9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:26.000 INFO 4486 --- [prometheus-minute-schedule-1,2faf2bb0aa31f0a0,3e94ab6c3aa57aa9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:26.462 INFO 4486 --- [ocp-schedules-12,ed4cf3a627bdb271,3c903af7e02f2662] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:23:26.462+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:23:26.464 INFO 4486 --- [ocp-schedules-12,ed4cf3a627bdb271,3c903af7e02f2662] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:23:26.465 INFO 4486 --- [ocp-schedules-12,ed4cf3a627bdb271,3c903af7e02f2662] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:23:26.465 INFO 4486 --- [ocp-schedules-12,ed4cf3a627bdb271,3c903af7e02f2662] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:23:26.637 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c9f8adc0ba60fda8,6b3a96fe7a965f14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c9f8adc0ba60fda8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:26.637 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c9f8adc0ba60fda8,6b3a96fe7a965f14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c9f8adc0ba60fda8, duration=1 ms] 2024-10-16 13:23:26.953 INFO 4486 --- [ocp-async-7,0dbd25d871a3cc7b,616193f1a4a02634] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:27.007 INFO 4486 --- [ocp-schedules-8,e21312ca2b04680b,69278f1bb97f8e80] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:23:27.046 WARN 4486 --- [scanner-13224160-1,e21312ca2b04680b,11f1ef3d36fe4ad0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:23:27.101 INFO 4486 --- [alarm-detect-11,e21312ca2b04680b,70fb6e93010ea038] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056180 ~ 1729056197 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056195, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056195, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056195=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:23:27.439 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:28.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bbfdf72446345663,4a69075e8a8ba37b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bbfdf72446345663, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:28.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bbfdf72446345663,4a69075e8a8ba37b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bbfdf72446345663, duration=1 ms] 2024-10-16 13:23:28.953 INFO 4486 --- [ocp-async-9,2a3d6e46684f555b,21fecee0a752c473] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:28.994 INFO 4486 --- [prometheus-minute-schedule-1,8c43276f9f36f516,f0619791cd985e82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:28.999 INFO 4486 --- [prometheus-minute-schedule-1,8c43276f9f36f516,f0619791cd985e82] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:29.947 INFO 4486 --- [ocp-schedules-28,fe63a88c7cc38f85,01fde56baac00633] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:23:30.103 INFO 4486 --- [prometheus-second-schedule-1,43119c6d27a85845,1954dca015b5ac04] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:23:30.326 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dfde894f989aa370,a6c74082698fff1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dfde894f989aa370, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:30.327 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dfde894f989aa370,a6c74082698fff1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dfde894f989aa370, duration=2 ms] 2024-10-16 13:23:30.440 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:30.912 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,366d991a33e3fd8a,31a53cf47ab4b97d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=366d991a33e3fd8a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:30.912 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,366d991a33e3fd8a,31a53cf47ab4b97d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=366d991a33e3fd8a, duration=1 ms] 2024-10-16 13:23:30.953 INFO 4486 --- [ocp-async-10,0980601059eb5a47,122c6ff37ebdd51b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:31.994 INFO 4486 --- [prometheus-minute-schedule-1,80ce44ea45dcdaa6,eb6161c4719afd00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:32.002 INFO 4486 --- [prometheus-minute-schedule-1,80ce44ea45dcdaa6,eb6161c4719afd00] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:32.786 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b05dc0d30642c4c5,6a2539776a9a7c87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=b05dc0d30642c4c5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:23:32.787 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b05dc0d30642c4c5,6a2539776a9a7c87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=b05dc0d30642c4c5, duration=3 ms] 2024-10-16 13:23:32.953 INFO 4486 --- [ocp-async-4,16ec47b966be0266,6d6d60df0f5e3524] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:32.969 INFO 4486 --- [ocp-async-7,15d2aa7a7e840f9b,0ff80f1a58371e54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:23:33.441 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:34.954 INFO 4486 --- [ocp-async-5,6540feb80062f640,e5f4faa8767ce1c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:34.994 INFO 4486 --- [prometheus-minute-schedule-1,d799b6d0f7eecdbb,e7a56e965abe4b65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:35.001 INFO 4486 --- [prometheus-minute-schedule-1,d799b6d0f7eecdbb,e7a56e965abe4b65] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:36.442 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:36.642 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,edd5f3b52576eec7,1015c48b1a4d4588] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=edd5f3b52576eec7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:36.643 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,edd5f3b52576eec7,1015c48b1a4d4588] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=edd5f3b52576eec7, duration=2 ms] 2024-10-16 13:23:36.954 INFO 4486 --- [ocp-async-10,dc6846d576e0832c,f8db70589bb3d798] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:37.063 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:23:37.064 INFO 4486 --- [ocp-schedules-24,2b9f5e2c0e2e7948,4f50cc07952ee800] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:23:37.066 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:23:37.066 INFO 4486 --- [ocp-schedules-22,e88489efc45148a5,29c0c51ea5b69804] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:23:37.068 INFO 4486 --- [ocp-schedules-22,e88489efc45148a5,29c0c51ea5b69804] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:23:37.068 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:23:37.069 INFO 4486 --- [ocp-schedules-24,2b9f5e2c0e2e7948,4f50cc07952ee800] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:23:37.081 INFO 4486 --- [ocp-schedules-12,e9e7a372e85b09b7,a24ec52d1bc27bbf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:23:37.083 INFO 4486 --- [ocp-schedules-12,e9e7a372e85b09b7,a24ec52d1bc27bbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:23:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:23:37.091 INFO 4486 --- [ocp-schedules-23,107c3a6c7ecdcb0e,044feb212ae4512b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:23:37.091 INFO 4486 --- [ocp-schedules-13,1914ed4e67c5c6e4,f3426bee6fa345a7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:23:37.092 INFO 4486 --- [ocp-schedules-17,2a5270369a835b34,b2b52b8f15f6e239] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:23:37.092 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:23:37.092 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:23:37.092 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:23:37.092 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:23:37.093 INFO 4486 --- [host-check-clockdiff1,2a5270369a835b34,c0ad2c7988ed475a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:23:37.094 INFO 4486 --- [host-check-clockdiff2,2a5270369a835b34,59ea53df3701105e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:23:37.094 INFO 4486 --- [host-check-clockdiff3,2a5270369a835b34,17b88796b6db3833] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:23:37.095 INFO 4486 --- [host-check-clockdiff4,2a5270369a835b34,f3f509835d12c6c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:23:37.095 INFO 4486 --- [host-check-agent-status1,1914ed4e67c5c6e4,73d5e20bfc3d4cd4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:23:37.096 INFO 4486 --- [host-check-agent-status2,1914ed4e67c5c6e4,0ce5210aafa13bf5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:23:37.097 INFO 4486 --- [host-check-agent-status3,1914ed4e67c5c6e4,71413423513b9657] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:23:37.098 INFO 4486 --- [host-check-agent-status4,1914ed4e67c5c6e4,5f1668a5f07be7d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:23:37.101 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:23:37.101 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:23:37.102 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:37.102 INFO 4486 --- [host-check-agent-status1,1914ed4e67c5c6e4,73d5e20bfc3d4cd4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:37.109 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:23:37.110 INFO 4486 --- [host-check-agent-status3,1914ed4e67c5c6e4,71413423513b9657] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:23:37.111 INFO 4486 --- [host-check-agent-status4,1914ed4e67c5c6e4,5f1668a5f07be7d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:23:37.111 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:23:37.111 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:23:37.111 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:37.111 INFO 4486 --- [host-check-agent-status1,1914ed4e67c5c6e4,73d5e20bfc3d4cd4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:37.111 INFO 4486 --- [host-check-agent-status2,1914ed4e67c5c6e4,0ce5210aafa13bf5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:23:37.111 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:23:37.112 INFO 4486 --- [host-check-agent-status3,1914ed4e67c5c6e4,71413423513b9657] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:23:37.122 INFO 4486 --- [host-check-agent-status4,1914ed4e67c5c6e4,5f1668a5f07be7d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:23:37.116 INFO 4486 --- [host-check-clockdiff1,2a5270369a835b34,c0ad2c7988ed475a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056217 0 0 , error= 2024-10-16 13:23:37.131 INFO 4486 --- [host-check-clockdiff4,2a5270369a835b34,f3f509835d12c6c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056217 3 3 , error= 2024-10-16 13:23:37.131 INFO 4486 --- [host-check-agent-status2,1914ed4e67c5c6e4,0ce5210aafa13bf5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:23:37.134 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.134 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:37.136 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:23:37.154 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.154 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:23:37.155 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:23:37.155 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:37.156 INFO 4486 --- [host-check-agent-status2,1914ed4e67c5c6e4,0ce5210aafa13bf5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:23:37.157 INFO 4486 --- [host-check-agent-status4,1914ed4e67c5c6e4,5f1668a5f07be7d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:23:37.157 INFO 4486 --- [host-check-agent-status1,1914ed4e67c5c6e4,73d5e20bfc3d4cd4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:23:37.158 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:23:37.158 INFO 4486 --- [host-check-agent-status3,1914ed4e67c5c6e4,71413423513b9657] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:23:37.159 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:23:37.160847451+08:00 2024-10-16 13:23:37.159 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:23:37.157648357+08:00 2024-10-16 13:23:37.159 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:23:37.159+08:00 2024-10-16 13:23:37.159 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:23:37.159+08:00 2024-10-16 13:23:37.160 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:23:37.160 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:23:37.163110615+08:00 2024-10-16 13:23:37.161 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:23:37.160+08:00 2024-10-16 13:23:37.162 INFO 4486 --- [ocp-schedules-7,7d783b8e721ff0f7,44444eda6e39370b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:23:37.162 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:23:37.159+08:00 and status AVAILABLE 2024-10-16 13:23:37.162 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:23:37.159+08:00 and status AVAILABLE 2024-10-16 13:23:37.162 INFO 4486 --- [host-check-agent-status3,1914ed4e67c5c6e4,71413423513b9657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:23:37.162 INFO 4486 --- [host-check-agent-status3,1914ed4e67c5c6e4,71413423513b9657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:23:37.163 INFO 4486 --- [host-check-agent-status3,1914ed4e67c5c6e4,71413423513b9657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:23:37.163 INFO 4486 --- [host-check-agent-status2,1914ed4e67c5c6e4,0ce5210aafa13bf5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:23:37.163 INFO 4486 --- [host-check-agent-status2,1914ed4e67c5c6e4,0ce5210aafa13bf5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:23:37.163 INFO 4486 --- [host-check-agent-status2,1914ed4e67c5c6e4,0ce5210aafa13bf5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:23:37.163 INFO 4486 --- [host-check-agent-status1,1914ed4e67c5c6e4,73d5e20bfc3d4cd4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:23:37.163 INFO 4486 --- [host-check-agent-status1,1914ed4e67c5c6e4,73d5e20bfc3d4cd4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:23:37.163 INFO 4486 --- [host-check-agent-status1,1914ed4e67c5c6e4,73d5e20bfc3d4cd4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:23:37.164 INFO 4486 --- [host-check-agent-status4,1914ed4e67c5c6e4,5f1668a5f07be7d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:23:37.164 INFO 4486 --- [host-check-agent-status4,1914ed4e67c5c6e4,5f1668a5f07be7d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:23:37.164 INFO 4486 --- [host-check-agent-status4,1914ed4e67c5c6e4,5f1668a5f07be7d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:23:37.164 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:23:37.160+08:00 and status AVAILABLE 2024-10-16 13:23:37.165 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:23:37.165 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:23:37.165 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:23:37.166 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:23:37.168909810+08:00 2024-10-16 13:23:37.167 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:23:37.166+08:00 2024-10-16 13:23:37.168 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:23:37.170 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:23:37.166+08:00 and status AVAILABLE 2024-10-16 13:23:37.172 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.173 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:23:37.175 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:37.176 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:23:37.177 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:23:37.177 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:37.178 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:23:37.179 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:23:37.180 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:23:37.183 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:23:37.184 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.185 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:23:37.185 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:23:37.193 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.194 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.194 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.195 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:23:37.195 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:23:37.205 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.205 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.205 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.205 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.205 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.205 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:37.205 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.206 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:37.208 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.210 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:37.210 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:23:37.213 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:23:37.219 WARN 4486 --- [scanner-13224374-1,7d783b8e721ff0f7,13fd78f2db495a1f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:23:37.225 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:23:37.234 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.235 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:23:37.239 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.240 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:23:37.242 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:23:37.245 INFO 4486 --- [alarm-detect-2,7d783b8e721ff0f7,821a2fd9ca35a8cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:37.250 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.251 INFO 4486 --- [ocp-schedules-1,fc00614b526b1641,fc47bfe6b70a2695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:23:37.251 INFO 4486 --- [alarm-detect-2,7d783b8e721ff0f7,821a2fd9ca35a8cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:37.242 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:23:37.253 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:23:37.258 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:23:37.259 INFO 4486 --- [alarm-detect-2,7d783b8e721ff0f7,821a2fd9ca35a8cd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:37.261 INFO 4486 --- [alarm-detect-20,7d783b8e721ff0f7,d23b28533a2e3c17] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:37.261 INFO 4486 --- [alarm-detect-18,7d783b8e721ff0f7,a5c7f48894f5c889] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:37.266 INFO 4486 --- [alarm-detect-20,7d783b8e721ff0f7,d23b28533a2e3c17] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:37.266 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:23:37.267 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:23:37.267 INFO 4486 --- [alarm-detect-18,7d783b8e721ff0f7,a5c7f48894f5c889] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:37.268 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:23:37.271 INFO 4486 --- [alarm-detect-20,7d783b8e721ff0f7,d23b28533a2e3c17] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:37.273 INFO 4486 --- [alarm-detect-18,7d783b8e721ff0f7,a5c7f48894f5c889] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:37.280 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:23:37.285 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:37.288 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:37.291 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:37.300 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:23:37.313 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:23:37.327 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:23:37.332 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:23:37.357 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:23:37.358 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:23:37.360 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:23:37.364 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:23:37.366 INFO 4486 --- [alarm-detect-6,7d783b8e721ff0f7,bafc6e62443d365f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:37.371 INFO 4486 --- [alarm-detect-6,7d783b8e721ff0f7,bafc6e62443d365f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:37.374 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:23:37.376 INFO 4486 --- [alarm-detect-6,7d783b8e721ff0f7,bafc6e62443d365f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:37.380 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:37.386 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:37.410 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:23:37.413 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:37.426 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:23:37.430 INFO 4486 --- [host-check-host-status3,107c3a6c7ecdcb0e,badf1b2ff44e4ed5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:23:37.437 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:23:37.447 INFO 4486 --- [host-check-host-status2,107c3a6c7ecdcb0e,41659f13f959ab98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:23:37.452 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:23:37.462 INFO 4486 --- [host-check-host-status4,107c3a6c7ecdcb0e,b68525ccdb357ce7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:23:37.486 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:23:37.498 INFO 4486 --- [host-check-host-status1,107c3a6c7ecdcb0e,ac6f2bc4bd173452] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:23:37.521 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:37.524 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.526 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:23:37.527 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.528 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.532 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:23:37.543 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.543 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.543 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.543 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.543 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.544 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.572 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:23:37.612 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:23:37.661 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:37.677 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:37.679 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.680 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:23:37.681 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:23:37.687 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.688 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:23:37.689 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.689 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.690 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:23:37.702 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.702 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.702 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.702 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.702 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.703 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.710 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:23:37.908 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:23:37.952 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:37.955 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.955 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:23:37.957 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.959 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.960 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:23:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:23:37.968 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.968 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.968 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.968 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.968 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:37.969 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:37.987 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:23:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:23:37.994 INFO 4486 --- [prometheus-minute-schedule-1,eff8f6dc4ffb2b69,741b67a73c17e678] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:38.000 INFO 4486 --- [prometheus-minute-schedule-1,eff8f6dc4ffb2b69,741b67a73c17e678] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:38.060 INFO 4486 --- [host-check-clockdiff3,2a5270369a835b34,17b88796b6db3833] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056218 3 3 , error= 2024-10-16 13:23:38.060 INFO 4486 --- [host-check-clockdiff2,2a5270369a835b34,59ea53df3701105e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056218 3 3 , error= 2024-10-16 13:23:38.213 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:23:38.259 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:38.261 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:38.262 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:23:38.263 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:38.264 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:38.264 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:23:38.279 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:38.279 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:38.279 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:38.279 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:38.279 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:23:38.280 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:38.344 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:23:38.425 INFO 4486 --- [ocp-schedules-1,2e9407b07f780adf,bfbed9c70d40de8a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:23:38.425 INFO 4486 --- [ocp-schedules-1,2e9407b07f780adf,bfbed9c70d40de8a] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:23:38.426 INFO 4486 --- [ocp-schedules-1,2e9407b07f780adf,bfbed9c70d40de8a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:23:38.461 INFO 4486 --- [ocp-schedules-7,0d60e899bc18bbfb,baa49d966d2c9aa8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8244} 2024-10-16 13:23:38.461 INFO 4486 --- [ocp-schedules-7,0d60e899bc18bbfb,baa49d966d2c9aa8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:23:38.539 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:23:38.346952, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:23:38.539 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:23:38.346952, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:23:38.578 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:23:38.346952} } 2024-10-16 13:23:38.646 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:23:38.652 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:23:38.655 INFO 4486 --- [ocp-schedules-5,b2ff083e73be6583,25eac4ba59e21513] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:23:38.716 INFO 4486 --- [ocp-schedules-23,e15317547e057aa9,464ad718d1e153ed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:23:38.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,36fdfcbb66782928,7d993a043f115e2d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=36fdfcbb66782928, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:38.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,36fdfcbb66782928,7d993a043f115e2d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=36fdfcbb66782928, duration=1 ms] 2024-10-16 13:23:38.954 INFO 4486 --- [ocp-async-7,cdddccf8489171f1,d520dc23bc29d308] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:39.443 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:39.953 INFO 4486 --- [ocp-schedules-4,8086215864fee833,7a08c548429ccad0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:23:40.006 INFO 4486 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:23:40.007 INFO 4486 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:23:40.031 INFO 4486 --- [Check obproxy2,f9d09d322707549f,1ab514b13466da72] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:23:40.031 INFO 4486 --- [Check obproxy2,f9d09d322707549f,1ab514b13466da72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:40.031 INFO 4486 --- [Check obproxy1,c1b826fb8708cf8f,82f2ee4b8d9148fd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:23:40.031 INFO 4486 --- [Check obproxy1,c1b826fb8708cf8f,82f2ee4b8d9148fd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:40.036 INFO 4486 --- [Collect obproxy1,2a554210932e8fad,2397e0a8679c0032] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:23:40.036 INFO 4486 --- [Collect obproxy1,2a554210932e8fad,2397e0a8679c0032] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:40.040 INFO 4486 --- [Collect obproxy2,9022364739f81d54,d44183cb56ef2249] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:23:40.040 INFO 4486 --- [Collect obproxy2,9022364739f81d54,d44183cb56ef2249] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:23:40.061 INFO 4486 --- [Check obproxy2,f9d09d322707549f,1ab514b13466da72] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:23:40.061 INFO 4486 --- [Check obproxy2,f9d09d322707549f,1ab514b13466da72] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:23:40.071 INFO 4486 --- [Check obproxy1,c1b826fb8708cf8f,82f2ee4b8d9148fd] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:23:40.071 INFO 4486 --- [Check obproxy1,c1b826fb8708cf8f,82f2ee4b8d9148fd] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:23:40.078 INFO 4486 --- [Collect obproxy2,9022364739f81d54,d44183cb56ef2249] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:23:40.084 INFO 4486 --- [Collect obproxy2,9022364739f81d54,d44183cb56ef2249] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:23:40.084 INFO 4486 --- [Collect obproxy1,2a554210932e8fad,2397e0a8679c0032] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:23:40.087 INFO 4486 --- [Collect obproxy1,2a554210932e8fad,2397e0a8679c0032] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:23:40.088 INFO 4486 --- [Collect obproxy2,9022364739f81d54,d44183cb56ef2249] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:23:40.090 INFO 4486 --- [Collect obproxy1,2a554210932e8fad,2397e0a8679c0032] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:23:40.332 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2e3fc6a26742ab71,7e3d85257bd8cfac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2e3fc6a26742ab71, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:40.333 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2e3fc6a26742ab71,7e3d85257bd8cfac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2e3fc6a26742ab71, duration=2 ms] 2024-10-16 13:23:40.828 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e810b098ea995187,e05389ed95a09a2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=e810b098ea995187, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:23:40.829 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e810b098ea995187,e05389ed95a09a2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=e810b098ea995187, duration=2 ms] 2024-10-16 13:23:40.916 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d2c4a55b0afd0bd7,6c180233cc91e9b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d2c4a55b0afd0bd7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:40.916 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d2c4a55b0afd0bd7,6c180233cc91e9b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d2c4a55b0afd0bd7, duration=1 ms] 2024-10-16 13:23:40.954 INFO 4486 --- [ocp-async-9,45b3e34c98ba7a26,1efb3955d2bab411] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:40.994 INFO 4486 --- [prometheus-minute-schedule-1,32c6b053107da306,00680e1e755ace9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:40.995 INFO 4486 --- [prometheus-hour-schedule-1,9e0861b4e3efd466,88e8218020a964b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:23:41.001 INFO 4486 --- [prometheus-minute-schedule-1,32c6b053107da306,00680e1e755ace9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:42.050 INFO 4486 --- [ocp-schedules-26,5d1efaf59a613a9d,51932d2c44031684] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:23:42.443 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:42.954 INFO 4486 --- [ocp-async-8,10eeed43fe37fe55,b6dbbe44b3aa2fec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:42.969 INFO 4486 --- [ocp-async-6,7a73a6514edf6a66,c082d187cd865d80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:23:43.994 INFO 4486 --- [prometheus-minute-schedule-1,db77241ff7b5f938,7adff49c3634535e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:44.000 INFO 4486 --- [prometheus-minute-schedule-1,db77241ff7b5f938,7adff49c3634535e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:44.955 INFO 4486 --- [ocp-async-4,30ae93815d2c000a,045fe66770a91f2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:45.386 INFO 4486 --- [ocp-schedules-2,0a4a22dffb1417e4,4fab5b67982d017a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:23:45.391 INFO 4486 --- [ocp-schedules-2,0a4a22dffb1417e4,4fab5b67982d017a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:23:45.391 INFO 4486 --- [ocp-schedules-2,0a4a22dffb1417e4,4fab5b67982d017a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:23:45.401 INFO 4486 --- [ocp-schedules-6,11b44b3a377d9e28,51c25c22fdd9301c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:23:45.401 INFO 4486 --- [ocp-schedules-6,11b44b3a377d9e28,51c25c22fdd9301c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:23:45.445 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:46.643 INFO 4486 --- [ocp-schedules-16,57ae5d81135ab87c,a1b8cfa5a95abfc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:23:46.644 INFO 4486 --- [ocp-schedules-16,f0797cc367f9d9bc,c7c8c61539fe7c34] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:23:46.645 INFO 4486 --- [ocp-schedules-16,f0797cc367f9d9bc,c7c8c61539fe7c34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:23:46.647 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f1e2e82eb1053076,be53eabc9c58d546] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f1e2e82eb1053076, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:46.647 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f1e2e82eb1053076,be53eabc9c58d546] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f1e2e82eb1053076, duration=1 ms] 2024-10-16 13:23:46.674 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.674 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:46.687 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.688 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:46.710 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:46.710 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.711 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.711 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:46.712 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:23:46.743 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:46.744 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.745 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:46.746 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:23:46.746 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.749 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:23:46.751 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:46.763 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:46.763 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:46.779 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:46.789 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:46.790 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.791 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:46.792 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:23:46.798 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:23:46.805 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:46.808 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:46.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:46.833 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.834 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:23:46.840 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:23:46.879 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:23:46.879 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:23:46.891 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:46.892 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.893 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:23:46.896 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.900 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:23:46.903 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:46.933 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:46.933 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:46.939 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:23:46.949 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:46.950 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:46.951 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:23:46.955 INFO 4486 --- [ocp-async-5,ec824416b04f5ec0,81b15f0e8da44150] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:46.961 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:23:46.961 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:23:46.961 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:46.962 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:46.988 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:46.988 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:46.994 INFO 4486 --- [prometheus-minute-schedule-1,6c02d2637664d17b,02783616c4f6acb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:47.001 INFO 4486 --- [prometheus-minute-schedule-1,6c02d2637664d17b,02783616c4f6acb3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:47.019 INFO 4486 --- [ocp-schedules-11,088289dddaf2b324,602bfd7ab6d5eb21] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:23:47.022 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:47.023 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:23:47.054 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:47.054 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:23:47.070 INFO 4486 --- [ocp-schedules-22,e88489efc45148a5,29c0c51ea5b69804] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:23:47.088 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:23:47.113 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:47.114 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:47.116 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:23:47.120 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:23:47.125 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:23:47.130 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:23:47.132 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:23:47.156 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:23:47.161 INFO 4486 --- [ocp-schedules-12,e9e7a372e85b09b7,a24ec52d1bc27bbf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:23:47.307 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:23:47.332 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:23:47.334 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:47.335 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:23:47.363 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:23:47.363 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055628392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:23:47.433 INFO 4486 --- [ocp-schedules-19,06d67bd7648ea2d1,7742fd788caf8da0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:23:47.460 WARN 4486 --- [scanner-13224706-1,06d67bd7648ea2d1,46f7a2e5a76418aa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:23:47.525 INFO 4486 --- [alarm-detect-18,06d67bd7648ea2d1,6d6a9c0a3a91f668] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056195 ~ 1729056217 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056210, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056210, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056210=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:23:48.359 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,11b31299d1c2c2f2,54ee1d45be7b6d67] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=11b31299d1c2c2f2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:23:48.359 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,11b31299d1c2c2f2,54ee1d45be7b6d67] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=11b31299d1c2c2f2, duration=1 ms] 2024-10-16 13:23:48.446 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:48.788 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:48.789 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:23:48.791 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:23:48.791 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:23:48.792 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:23:48.840 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d2955609b06b3037,728fb2aaaa55bba3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d2955609b06b3037, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:48.840 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d2955609b06b3037,728fb2aaaa55bba3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d2955609b06b3037, duration=1 ms] 2024-10-16 13:23:48.955 INFO 4486 --- [ocp-async-8,1e247aa9d96edb0b,2caaedf8f9594284] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:49.323 INFO 4486 --- [ocp-schedules-14,e4147698d41a4b00,163d0f5b740b2731] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:23:49.323 INFO 4486 --- [ocp-schedules-14,e4147698d41a4b00,163d0f5b740b2731] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:23:49.323 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:23:49.323 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:23:49.323 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:23:49.323 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:23:49.323 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:23:49.328 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:23:49.329 INFO 4486 --- [ocp-schedules-14,e4147698d41a4b00,163d0f5b740b2731] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:23:49.329 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:23:49.329 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:23:49.330 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:23:49.330 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:23:49.330 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:23:49.330 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:23:49.331 INFO 4486 --- [ocp-schedules-14,e4147698d41a4b00,163d0f5b740b2731] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:23:49.331 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:23:49.331 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:23:49.332 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:23:49.354 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:23:49.354 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:23:49.355 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:23:49.357 INFO 4486 --- [ocp-schedules-14,e4147698d41a4b00,163d0f5b740b2731] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:23:49.357 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:23:49.362 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:23:49.367 INFO 4486 --- [ocp-schedules-14,e4147698d41a4b00,163d0f5b740b2731] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:23:49.958 INFO 4486 --- [ocp-schedules-5,492fe00e4301ac7f,30340dfdf05aa351] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:23:49.994 INFO 4486 --- [prometheus-minute-schedule-1,9d4bb014311ac269,f4ef7267ee3e6dce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:49.999 INFO 4486 --- [prometheus-minute-schedule-1,9d4bb014311ac269,f4ef7267ee3e6dce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:50.300 INFO 4486 --- [ocp-schedules-12,db30a7630413c113,ef617a1ad792c623] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:23:50.300 INFO 4486 --- [ocp-schedules-12,db30a7630413c113,ef617a1ad792c623] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:23:50.305 INFO 4486 --- [ocp-schedules-12,db30a7630413c113,ef617a1ad792c623] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:23:50.305 INFO 4486 --- [ocp-schedules-12,db30a7630413c113,ef617a1ad792c623] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:23:50.308 INFO 4486 --- [ocp-schedules-12,db30a7630413c113,ef617a1ad792c623] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:23:50.308 INFO 4486 --- [ocp-schedules-12,db30a7630413c113,ef617a1ad792c623] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703033600770641920 2024-10-16 13:23:50.338 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c4fc08905ea00858,032412ab74128831] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c4fc08905ea00858, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:50.338 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c4fc08905ea00858,032412ab74128831] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c4fc08905ea00858, duration=1 ms] 2024-10-16 13:23:50.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,744ab8e611221667,5391abd338569052] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=744ab8e611221667, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:50.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,744ab8e611221667,5391abd338569052] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=744ab8e611221667, duration=1 ms] 2024-10-16 13:23:50.955 INFO 4486 --- [ocp-async-2,3af0562777a0b24c,2c9d013b47e37c58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:51.447 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:52.034 INFO 4486 --- [ocp-schedules-29,852fa4643609f48b,6711f93fc69e8f0b] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:23:52.955 INFO 4486 --- [ocp-async-3,19882d950ec97a25,f05d2258c23d7557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:52.969 INFO 4486 --- [ocp-async-9,2448dfa57a228886,d6f61a75d73b072c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:23:52.994 INFO 4486 --- [prometheus-minute-schedule-1,8523dd546017be37,cff0eaa5467f523f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:53.000 INFO 4486 --- [prometheus-minute-schedule-1,8523dd546017be37,cff0eaa5467f523f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:53.421 INFO 4486 --- [ocp-schedules-23,2b579cf998440233,6ccf0f2d05ecb784] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:23:53.423 INFO 4486 --- [ocp-schedules-23,2b579cf998440233,6ccf0f2d05ecb784] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:23:53.424 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:23:53.424 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:23:53.424 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:23:54.283 INFO 4486 --- [ocp-schedules-21,a629c217d53c45a7,84084d6f58b9d4a5] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:23:54.283+08:00, tenantSize=5 2024-10-16 13:23:54.288 INFO 4486 --- [ocp-schedules-21,a629c217d53c45a7,84084d6f58b9d4a5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729056234284000 2024-10-16 13:23:54.288 INFO 4486 --- [ocp-schedules-21,a629c217d53c45a7,84084d6f58b9d4a5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729055934217000 2024-10-16 13:23:54.306 INFO 4486 --- [ocp-schedules-21,a629c217d53c45a7,84084d6f58b9d4a5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:23:54.311 INFO 4486 --- [ocp-schedules-21,a629c217d53c45a7,84084d6f58b9d4a5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729056234308000 2024-10-16 13:23:54.311 INFO 4486 --- [ocp-schedules-21,a629c217d53c45a7,84084d6f58b9d4a5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729055934243000 2024-10-16 13:23:54.342 INFO 4486 --- [ocp-schedules-21,a629c217d53c45a7,84084d6f58b9d4a5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:23:54.448 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:54.956 INFO 4486 --- [ocp-async-10,4cdabb292f54d792,35611c2b4b314862] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:55.994 INFO 4486 --- [prometheus-minute-schedule-1,9320d22cc97fc972,479219aafc0f814f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:56.000 INFO 4486 --- [prometheus-minute-schedule-1,9320d22cc97fc972,479219aafc0f814f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:56.657 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1a2f857496ead303,bbb85669ece4ed94] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1a2f857496ead303, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:56.657 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1a2f857496ead303,bbb85669ece4ed94] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1a2f857496ead303, duration=1 ms] 2024-10-16 13:23:56.956 INFO 4486 --- [ocp-async-2,b444f920f0558df9,d95ea298b5184450] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:57.448 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:23:57.568 INFO 4486 --- [ocp-schedules-27,1141943239c5b8c3,7e7a5e87ed1bcc03] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:23:57.625 WARN 4486 --- [scanner-13224949-1,1141943239c5b8c3,69dbb814edaeed44] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:23:57.693 INFO 4486 --- [alarm-detect-20,1141943239c5b8c3,7979cac02e81d273] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:57.705 INFO 4486 --- [alarm-detect-20,1141943239c5b8c3,7979cac02e81d273] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:57.713 INFO 4486 --- [alarm-detect-20,1141943239c5b8c3,7979cac02e81d273] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:23:57.721 INFO 4486 --- [alarm-detect-14,1141943239c5b8c3,af2bcc8ccd245ce2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:57.730 INFO 4486 --- [alarm-detect-5,1141943239c5b8c3,35d1eb9f19cda1dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:57.731 INFO 4486 --- [alarm-detect-14,1141943239c5b8c3,af2bcc8ccd245ce2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:57.735 INFO 4486 --- [alarm-detect-14,1141943239c5b8c3,af2bcc8ccd245ce2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:23:57.736 INFO 4486 --- [alarm-detect-5,1141943239c5b8c3,35d1eb9f19cda1dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:57.734 INFO 4486 --- [alarm-detect-13,1141943239c5b8c3,9cd50cc6178d52c2] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056210 ~ 1729056227 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056225, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056225, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056225=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:23:57.741 INFO 4486 --- [alarm-detect-5,1141943239c5b8c3,35d1eb9f19cda1dc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:23:57.820 INFO 4486 --- [alarm-detect-17,1141943239c5b8c3,1904c5e4cda683cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:57.828 INFO 4486 --- [alarm-detect-17,1141943239c5b8c3,1904c5e4cda683cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:57.836 INFO 4486 --- [alarm-detect-17,1141943239c5b8c3,1904c5e4cda683cf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:23:58.720 INFO 4486 --- [ocp-schedules-30,8cd8d5aa18c66095,5518d5176c2caf4c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:23:58.844 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,14f5da2199f2b6fa,8b6899382fb981c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=14f5da2199f2b6fa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:23:58.845 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,14f5da2199f2b6fa,8b6899382fb981c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=14f5da2199f2b6fa, duration=1 ms] 2024-10-16 13:23:58.956 INFO 4486 --- [ocp-async-5,5a0b73aacb2954e3,224161bce044d799] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:23:58.994 INFO 4486 --- [prometheus-minute-schedule-1,14a461dadda16693,d288986139b845b7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056240, interval=60 2024-10-16 13:23:59.000 INFO 4486 --- [prometheus-minute-schedule-1,14a461dadda16693,d288986139b845b7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:23:59.964 INFO 4486 --- [ocp-schedules-24,035ed8e234a9dd80,0b6acd46be1e927b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:24:00.134 INFO 4486 --- [ocp-schedules-29,7a60fc7fd1f9c99f,100240638d57e875] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:24:00.134+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:24:00.137 INFO 4486 --- [ocp-schedules-29,7a60fc7fd1f9c99f,100240638d57e875] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:24:00.193 INFO 4486 --- [ocp-schedules-29,7a60fc7fd1f9c99f,100240638d57e875] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for obce_3zones, save 1 to SqlAggregateData 2024-10-16 13:24:00.294 INFO 4486 --- [ocp-schedules-29,7a60fc7fd1f9c99f,100240638d57e875] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for glob_3zone, save 1 to SqlAggregateData 2024-10-16 13:24:00.342 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,35c7c8cbccd7e234,a02b863fd3d9b427] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=35c7c8cbccd7e234, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:00.342 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,35c7c8cbccd7e234,a02b863fd3d9b427] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=35c7c8cbccd7e234, duration=1 ms] 2024-10-16 13:24:00.449 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:00.926 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2b393e32cc21e3e6,d3f38744c2edb7f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2b393e32cc21e3e6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:00.926 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2b393e32cc21e3e6,d3f38744c2edb7f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2b393e32cc21e3e6, duration=1 ms] 2024-10-16 13:24:00.956 INFO 4486 --- [ocp-async-8,c53712883ce0e509,577921d25f79567c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:01.847 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:24:01.847 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=7 2024-10-16 13:24:01.847 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2024-10-16 13:24:01.847 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1000015 2024-10-16 13:24:01.848 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2024-10-16 13:24:01.871 WARN 4486 --- [ForkJoinPool.commonPool-worker-13,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:24:01.871 WARN 4486 --- [ForkJoinPool.commonPool-worker-30,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:24:01.886 WARN 4486 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.928 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:24:01.929 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 7 agent sql collect property 2024-10-16 13:24:01.929 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:01.933 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:01.956 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2024-10-16 13:24:01.963 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:24:01.963 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:24:01.964 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:24:01.965 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2024-10-16 13:24:01.965 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1000015 agent sql collect property 2024-10-16 13:24:01.967 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2024-10-16 13:24:01.970 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:01.973 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:01.973 INFO 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 7, do nothing 2024-10-16 13:24:01.977 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2024-10-16 13:24:01.978 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:01.980 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:01.981 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:01.984 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:01.994 INFO 4486 --- [prometheus-minute-schedule-1,239f567d15bb30c7,74413bfb71a3a1db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:02.000 INFO 4486 --- [prometheus-minute-schedule-1,239f567d15bb30c7,74413bfb71a3a1db] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:02.006 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313639), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.178), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data), (monagent.cluster.id=5), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=obce_3zones), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=2), (monagent.obproxy.cluster.name=obce_3zones), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=10.197.49.178), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLev 2024-10-16 13:24:02.015 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313644), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=172.28.46.152), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1000003), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=glob_3zone), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=1000002), (monagent.obproxy.cluster.name=xc_OB43), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=172.28.46.152), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition fro 2024-10-16 13:24:02.017 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313637), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.184), (ob.run.path=/root/oceanbase), (ob.install.path=/root/oceanbase), (ob.data.path=/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=OceanBase), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local ser 2024-10-16 13:24:02.023 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2024-10-16 13:24:02.025 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1000015, do nothing 2024-10-16 13:24:02.058 ERROR 4486 --- [ocp-schedules-11,dc220474c001277a,ddb588cc754b907b] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor1649.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:Unexpected error: key monagent.ob.cgroup.status is not found in config properties! at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:476) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:286) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:114) at com.oceanbase.ocp.executor.executor.AgentExecutor.updateAgentConfig(AgentExecutor.java:193) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentProperties(HostAgentConfigManagerImpl.java:188) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentConfig(HostAgentConfigManagerImpl.java:196) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$FastClassBySpringCGLIB$$798eb4cd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$EnhancerBySpringCGLIB$$eb50b66a.refreshAgentConfig() at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.refreshHostAgentConfig(HostAgentServiceImpl.java:734) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$5f28aa5.refreshHostAgentConfig() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$refreshAgentConfig$0(HostSchedules.java:92) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:24:02.957 INFO 4486 --- [ocp-async-4,100a2fcc00e84c91,1523d8ac66d88cad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:02.970 INFO 4486 --- [ocp-async-2,ee5d6d06d4a314d5,84b79a534c785808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:24:03.425 INFO 4486 --- [ocp-schedules-23,2b579cf998440233,6ccf0f2d05ecb784] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:24:03.450 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:04.957 INFO 4486 --- [ocp-async-3,ae507efda83f712f,03db280fdc4f871e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:04.994 INFO 4486 --- [prometheus-minute-schedule-1,4c4afcdb05f5ec2f,912262782bc329ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:05.003 INFO 4486 --- [prometheus-minute-schedule-1,4c4afcdb05f5ec2f,912262782bc329ca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:06.451 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:06.668 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,18a1c9f1f3a4aa71,c4f4cabb2d512c1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=18a1c9f1f3a4aa71, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:06.668 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,18a1c9f1f3a4aa71,c4f4cabb2d512c1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=18a1c9f1f3a4aa71, duration=2 ms] 2024-10-16 13:24:06.957 INFO 4486 --- [ocp-async-8,aba32c0fd595e8ff,729443ff609cf1ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:07.066 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:24:07.067 INFO 4486 --- [ocp-schedules-23,82ee3ba3105acfe4,2f01da086b0a9bf3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:24:07.069 INFO 4486 --- [ocp-schedules-23,82ee3ba3105acfe4,2f01da086b0a9bf3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:07.081 INFO 4486 --- [ocp-schedules-1,ae8c2a3c876eab6d,b0b23455a3828a9a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:24:07.083 INFO 4486 --- [ocp-schedules-1,ae8c2a3c876eab6d,b0b23455a3828a9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:24:07.089 INFO 4486 --- [ocp-schedules-10,ed06ce5eb0d16d7f,081f2d1ad6ba1b66] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:24:07.090 INFO 4486 --- [ocp-schedules-20,615ed56cc270d123,330da1c434bd136a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:24:07.090 INFO 4486 --- [host-check-clockdiff1,ed06ce5eb0d16d7f,c4db695d963a449d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:24:07.090 INFO 4486 --- [host-check-clockdiff2,ed06ce5eb0d16d7f,c8419a9fe3bcd026] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:24:07.091 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:24:07.091 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:24:07.091 INFO 4486 --- [host-check-clockdiff3,ed06ce5eb0d16d7f,8508197c2db95a48] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:24:07.091 INFO 4486 --- [ocp-schedules-11,5208bc168ea447c9,d45dd3594e1a1c6a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:24:07.092 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:24:07.092 INFO 4486 --- [host-check-clockdiff4,ed06ce5eb0d16d7f,6813285fdeb5825e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:24:07.094 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:24:07.095 INFO 4486 --- [host-check-agent-status1,5208bc168ea447c9,30be0859ae49729f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:24:07.095 INFO 4486 --- [host-check-agent-status2,5208bc168ea447c9,429e2cb6dcc4e523] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:24:07.096 INFO 4486 --- [host-check-agent-status4,5208bc168ea447c9,53a3c187e21cac21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:24:07.096 INFO 4486 --- [host-check-agent-status3,5208bc168ea447c9,ff29871a495c869c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:24:07.102 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:07.102 INFO 4486 --- [host-check-agent-status3,5208bc168ea447c9,ff29871a495c869c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:07.102 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:07.103 INFO 4486 --- [host-check-agent-status2,5208bc168ea447c9,429e2cb6dcc4e523] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:07.103 INFO 4486 --- [host-check-agent-status1,5208bc168ea447c9,30be0859ae49729f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:07.103 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:07.138 INFO 4486 --- [host-check-agent-status4,5208bc168ea447c9,53a3c187e21cac21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:07.138 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:07.138 INFO 4486 --- [host-check-agent-status2,5208bc168ea447c9,429e2cb6dcc4e523] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:07.140 INFO 4486 --- [host-check-clockdiff2,ed06ce5eb0d16d7f,c8419a9fe3bcd026] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056247 3 3 , error= 2024-10-16 13:24:07.142 INFO 4486 --- [host-check-clockdiff1,ed06ce5eb0d16d7f,c4db695d963a449d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056247 0 0 , error= 2024-10-16 13:24:07.142 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:07.142 INFO 4486 --- [host-check-agent-status1,5208bc168ea447c9,30be0859ae49729f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:07.144 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:07.144 INFO 4486 --- [host-check-agent-status3,5208bc168ea447c9,ff29871a495c869c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:07.145 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:07.145 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.146 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:07.146 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:07.146 INFO 4486 --- [host-check-agent-status4,5208bc168ea447c9,53a3c187e21cac21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:07.147 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:24:07.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:07.171 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:24:07.171 INFO 4486 --- [host-check-agent-status3,5208bc168ea447c9,ff29871a495c869c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:24:07.172 INFO 4486 --- [host-check-agent-status2,5208bc168ea447c9,429e2cb6dcc4e523] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:24:07.173 INFO 4486 --- [host-check-agent-status1,5208bc168ea447c9,30be0859ae49729f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:24:07.173 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:24:07.174 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:24:07.174 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:24:07.174344404+08:00 2024-10-16 13:24:07.175 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:24:07.175 INFO 4486 --- [host-check-agent-status4,5208bc168ea447c9,53a3c187e21cac21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:24:07.175 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:24:07.178113454+08:00 2024-10-16 13:24:07.175 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:24:07.175+08:00 2024-10-16 13:24:07.176 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:24:07.175+08:00 2024-10-16 13:24:07.176 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:24:07.179067261+08:00 2024-10-16 13:24:07.176 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:24:07.177 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:24:07.176+08:00 2024-10-16 13:24:07.177 INFO 4486 --- [host-check-agent-status3,5208bc168ea447c9,ff29871a495c869c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:24:07.177 INFO 4486 --- [host-check-agent-status3,5208bc168ea447c9,ff29871a495c869c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:24:07.177 INFO 4486 --- [host-check-agent-status3,5208bc168ea447c9,ff29871a495c869c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:24:07.178 INFO 4486 --- [host-check-agent-status1,5208bc168ea447c9,30be0859ae49729f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:24:07.178 INFO 4486 --- [host-check-agent-status1,5208bc168ea447c9,30be0859ae49729f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:24:07.178 INFO 4486 --- [host-check-agent-status1,5208bc168ea447c9,30be0859ae49729f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:24:07.178 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:24:07.175+08:00 and status AVAILABLE 2024-10-16 13:24:07.178 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:24:07.175+08:00 and status AVAILABLE 2024-10-16 13:24:07.179 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:24:07.176+08:00 and status AVAILABLE 2024-10-16 13:24:07.179 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:24:07.181129602+08:00 2024-10-16 13:24:07.179 INFO 4486 --- [host-check-agent-status2,5208bc168ea447c9,429e2cb6dcc4e523] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:24:07.179 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:24:07.179+08:00 2024-10-16 13:24:07.179 INFO 4486 --- [host-check-agent-status2,5208bc168ea447c9,429e2cb6dcc4e523] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:24:07.179 INFO 4486 --- [host-check-agent-status2,5208bc168ea447c9,429e2cb6dcc4e523] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:24:07.179 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.180 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:24:07.180 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:24:07.181 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:24:07.182 INFO 4486 --- [host-check-agent-status4,5208bc168ea447c9,53a3c187e21cac21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:24:07.182 INFO 4486 --- [host-check-agent-status4,5208bc168ea447c9,53a3c187e21cac21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:24:07.182 INFO 4486 --- [host-check-agent-status4,5208bc168ea447c9,53a3c187e21cac21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:24:07.183 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:24:07.179+08:00 and status AVAILABLE 2024-10-16 13:24:07.184 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:24:07.185 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:24:07.189 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:07.189 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:07.189 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:07.191 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:07.191 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:07.191 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:07.194 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.194 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:07.195 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:07.195 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:07.211 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.212 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:07.215 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:07.215 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:07.218 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:07.218 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:07.227 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.228 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:07.230 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.231 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:24:07.240 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:07.241 INFO 4486 --- [ocp-schedules-15,211bb6f1a9f41592,0eb5019831695b25] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:24:07.246 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:24:07.253 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:24:07.257 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:24:07.258 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:24:07.264 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:24:07.269 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:24:07.275 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:07.277 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:24:07.283 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:07.286 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:07.287 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:24:07.300 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:07.313 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:24:07.324 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:24:07.331 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:24:07.333 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:24:07.340 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:07.342 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:24:07.344 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:24:07.351 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:24:07.360 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:07.360 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:07.361 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:24:07.368 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:24:07.378 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:07.380 INFO 4486 --- [host-check-host-status3,615ed56cc270d123,53707b70a52c5f17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:24:07.397 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:24:07.397 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:24:07.408 INFO 4486 --- [host-check-host-status2,615ed56cc270d123,198a33fa6576c6ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:24:07.410 INFO 4486 --- [host-check-host-status4,615ed56cc270d123,941033fd8cc75d5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:24:07.430 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:24:07.442 INFO 4486 --- [host-check-host-status1,615ed56cc270d123,0ee10011472730f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:24:07.892 INFO 4486 --- [ocp-schedules-24,90df960685b99d94,9faaf2573b3d0649] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:24:07.941 WARN 4486 --- [scanner-13225297-1,90df960685b99d94,10fb4d02a4817e63] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:24:07.995 INFO 4486 --- [prometheus-minute-schedule-1,722d8182dfc324f4,69cb2bf2465fab65] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:08.015 INFO 4486 --- [prometheus-minute-schedule-1,722d8182dfc324f4,69cb2bf2465fab65] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:08.068 INFO 4486 --- [host-check-clockdiff3,ed06ce5eb0d16d7f,8508197c2db95a48] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056248 3 3 , error= 2024-10-16 13:24:08.073 INFO 4486 --- [host-check-clockdiff4,ed06ce5eb0d16d7f,6813285fdeb5825e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056248 2 2 , error= 2024-10-16 13:24:08.851 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,148b5f3a9ad5d6bd,4143ae5e35c6d21f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=148b5f3a9ad5d6bd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:08.851 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,148b5f3a9ad5d6bd,4143ae5e35c6d21f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=148b5f3a9ad5d6bd, duration=1 ms] 2024-10-16 13:24:08.957 INFO 4486 --- [ocp-async-6,27e33ccac0c53d35,71a03140a190278d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:09.453 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:09.969 INFO 4486 --- [ocp-schedules-5,57e953943b2135e3,d56677158b9394a2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:24:10.007 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:24:10.007 INFO 4486 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:24:10.028 INFO 4486 --- [Check obproxy2,66c6072a4b7230e1,75f3ac152473d862] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:24:10.028 INFO 4486 --- [Check obproxy1,9f516787984060d2,095cd0d3e64ba590] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:24:10.028 INFO 4486 --- [Check obproxy1,9f516787984060d2,095cd0d3e64ba590] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:10.028 INFO 4486 --- [Check obproxy2,66c6072a4b7230e1,75f3ac152473d862] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:10.030 INFO 4486 --- [Collect obproxy1,647cb120ccc3136e,673950ceac5587ab] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:24:10.031 INFO 4486 --- [Collect obproxy1,647cb120ccc3136e,673950ceac5587ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:10.035 INFO 4486 --- [Collect obproxy2,9acdb382dc22455d,0c86c712b9f821d6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:24:10.035 INFO 4486 --- [Collect obproxy2,9acdb382dc22455d,0c86c712b9f821d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:10.052 INFO 4486 --- [Check obproxy2,66c6072a4b7230e1,75f3ac152473d862] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:24:10.052 INFO 4486 --- [Check obproxy2,66c6072a4b7230e1,75f3ac152473d862] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:24:10.053 INFO 4486 --- [Collect obproxy1,647cb120ccc3136e,673950ceac5587ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:24:10.056 INFO 4486 --- [Collect obproxy2,9acdb382dc22455d,0c86c712b9f821d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:24:10.056 INFO 4486 --- [Collect obproxy1,647cb120ccc3136e,673950ceac5587ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:24:10.058 INFO 4486 --- [Collect obproxy1,647cb120ccc3136e,673950ceac5587ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:24:10.059 INFO 4486 --- [Check obproxy1,9f516787984060d2,095cd0d3e64ba590] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:24:10.059 INFO 4486 --- [Check obproxy1,9f516787984060d2,095cd0d3e64ba590] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:24:10.061 INFO 4486 --- [Collect obproxy2,9acdb382dc22455d,0c86c712b9f821d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:24:10.065 INFO 4486 --- [Collect obproxy2,9acdb382dc22455d,0c86c712b9f821d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:24:10.346 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1796a22a1e148861,6bbe1409d537c064] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1796a22a1e148861, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:10.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1796a22a1e148861,6bbe1409d537c064] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1796a22a1e148861, duration=2 ms] 2024-10-16 13:24:10.931 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,df67ffcc7765594d,59e4c22ff235255d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=df67ffcc7765594d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:10.932 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,df67ffcc7765594d,59e4c22ff235255d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=df67ffcc7765594d, duration=2 ms] 2024-10-16 13:24:10.957 INFO 4486 --- [ocp-async-2,d5d83281aa564dbe,2cf9cc81b8deb930] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:10.980 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:24:10.994 INFO 4486 --- [prometheus-minute-schedule-1,fc5b001b040783a4,dbfce0ce5cdee3de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:11.001 INFO 4486 --- [prometheus-minute-schedule-1,fc5b001b040783a4,dbfce0ce5cdee3de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:11.013 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:24:12.454 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:12.781 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056180, nodeCount=68, affectRows=68 2024-10-16 13:24:12.782 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056180, nodeCount=68, validCount=68 2024-10-16 13:24:12.788 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056180, nodeCount=191, affectRows=191 2024-10-16 13:24:12.789 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056180, nodeCount=189, validCount=189 2024-10-16 13:24:12.796 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056180, nodeCount=192, affectRows=192 2024-10-16 13:24:12.797 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056180, nodeCount=192, validCount=192 2024-10-16 13:24:12.805 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056180, nodeCount=1000, affectRows=1000 2024-10-16 13:24:12.806 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056180, nodeCount=1000, affectRows=1000 2024-10-16 13:24:12.808 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056180, nodeCount=1000, affectRows=1000 2024-10-16 13:24:12.810 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056180, nodeCount=1000, validCount=1000 2024-10-16 13:24:12.861 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056180, nodeCount=999, validCount=999 2024-10-16 13:24:12.862 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056180, nodeCount=1000, validCount=1000 2024-10-16 13:24:12.958 INFO 4486 --- [ocp-async-9,4665563c4c987982,8593de3d3885f07d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:12.970 INFO 4486 --- [ocp-async-1,31d153d831fff899,54c6767edd2e4d22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:24:13.994 INFO 4486 --- [prometheus-minute-schedule-1,395f9b869af3850e,0b6aff1ea21a9c8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:14.001 INFO 4486 --- [prometheus-minute-schedule-1,395f9b869af3850e,0b6aff1ea21a9c8c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:14.211 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056180, nodeCount=477, affectRows=477 2024-10-16 13:24:14.899 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056180, nodeCount=477, validCount=477 2024-10-16 13:24:14.958 INFO 4486 --- [ocp-async-10,2984323c4176759b,b0f3e27319eeaee6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:15.455 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:16.553 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bf6e148309dad625,d7fd8fb74566b0af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=bf6e148309dad625, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:24:16.553 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bf6e148309dad625,d7fd8fb74566b0af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=bf6e148309dad625, duration=2 ms] 2024-10-16 13:24:16.674 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2abd3bc6107b0ab4,bdfcae3b2afd1572] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2abd3bc6107b0ab4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:16.674 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2abd3bc6107b0ab4,bdfcae3b2afd1572] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2abd3bc6107b0ab4, duration=1 ms] 2024-10-16 13:24:16.958 INFO 4486 --- [ocp-async-5,737beed17ab45a24,7925ab9681f9a21d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:16.994 INFO 4486 --- [prometheus-minute-schedule-1,d14fee19201d7e0a,24d6d8df89c60eaa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:17.000 INFO 4486 --- [prometheus-minute-schedule-1,d14fee19201d7e0a,24d6d8df89c60eaa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:17.022 INFO 4486 --- [ocp-schedules-15,0f387cf4fdb79b12,c64ba74425196de1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:24:17.072 INFO 4486 --- [ocp-schedules-23,82ee3ba3105acfe4,2f01da086b0a9bf3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:24:17.175 INFO 4486 --- [ocp-schedules-1,ae8c2a3c876eab6d,b0b23455a3828a9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:24:17.364 INFO 4486 --- [ocp-schedules-16,f0797cc367f9d9bc,c7c8c61539fe7c34] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:24:17.365 INFO 4486 --- [ocp-schedules-16,d475b2fd4edf4180,cd01f1aa4f95ba44] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:24:17.366 INFO 4486 --- [ocp-schedules-16,d475b2fd4edf4180,cd01f1aa4f95ba44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:17.390 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.391 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:17.403 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.404 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:17.427 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.428 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.428 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.429 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:24:17.429 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:17.455 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.456 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.456 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:17.457 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:24:17.457 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.460 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:24:17.461 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:17.470 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.470 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:17.480 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.490 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.490 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.491 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:17.492 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:24:17.499 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:24:17.504 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:17.506 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:17.524 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.525 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.526 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:24:17.530 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:24:17.557 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:24:17.557 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:24:17.573 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.575 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.576 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:17.580 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.584 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:24:17.587 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:17.603 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:24:17.611 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.612 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:17.612 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.613 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.614 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:24:17.622 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:24:17.622 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:24:17.637 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.637 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:17.662 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.662 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:17.692 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.692 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:17.717 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.717 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:17.746 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:17.773 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.775 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.776 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:17.780 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:24:17.785 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:24:17.789 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:24:17.791 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:24:17.812 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:24:17.957 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:24:17.983 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:17.985 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:17.986 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:24:18.015 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:24:18.015 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:24:18.082 INFO 4486 --- [ocp-schedules-17,844e939aa21fe1f5,a10c8d4ba4f4c5a8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:24:18.119 WARN 4486 --- [scanner-13226336-1,844e939aa21fe1f5,461184a35b52e63d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:24:18.137 INFO 4486 --- [alarm-detect-9,844e939aa21fe1f5,bf2115ad08f5628c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004833, data is empty. 2024-10-16 13:24:18.142 INFO 4486 --- [alarm-detect-9,844e939aa21fe1f5,bf2115ad08f5628c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:18.147 INFO 4486 --- [alarm-detect-9,844e939aa21fe1f5,bf2115ad08f5628c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:18.159 INFO 4486 --- [alarm-detect-14,844e939aa21fe1f5,58c0b4b4b668fd45] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004833, data is empty. 2024-10-16 13:24:18.159 INFO 4486 --- [alarm-detect-12,844e939aa21fe1f5,501d3a19f6ef8e7f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004833, data is empty. 2024-10-16 13:24:18.165 INFO 4486 --- [alarm-detect-14,844e939aa21fe1f5,58c0b4b4b668fd45] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:18.165 INFO 4486 --- [alarm-detect-12,844e939aa21fe1f5,501d3a19f6ef8e7f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:18.170 INFO 4486 --- [alarm-detect-14,844e939aa21fe1f5,58c0b4b4b668fd45] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:18.170 INFO 4486 --- [alarm-detect-12,844e939aa21fe1f5,501d3a19f6ef8e7f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:18.191 INFO 4486 --- [alarm-detect-20,844e939aa21fe1f5,968607702f7100f9] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056225 ~ 1729056248 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056240, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056240, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056240=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:24:18.229 INFO 4486 --- [alarm-detect-3,844e939aa21fe1f5,4528828a4a52d3c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004833, data is empty. 2024-10-16 13:24:18.233 INFO 4486 --- [alarm-detect-3,844e939aa21fe1f5,4528828a4a52d3c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:18.237 INFO 4486 --- [alarm-detect-3,844e939aa21fe1f5,4528828a4a52d3c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:18.456 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:18.724 INFO 4486 --- [ocp-schedules-29,659dd79477f7aabf,ebc5d14b66fa48d0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:24:18.855 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e5c94cc2abeb66e2,06d64744a825f4dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e5c94cc2abeb66e2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:18.855 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e5c94cc2abeb66e2,06d64744a825f4dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e5c94cc2abeb66e2, duration=1 ms] 2024-10-16 13:24:18.958 INFO 4486 --- [ocp-async-9,39b71f9c5ce148b8,f07e821c377b5d64] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:19.974 INFO 4486 --- [ocp-schedules-13,1d93d0e73892d74c,c4103be21a24ae5f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:24:19.994 INFO 4486 --- [prometheus-minute-schedule-1,ac52da994cd9c830,0395eb3152abc9f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:19.999 INFO 4486 --- [prometheus-minute-schedule-1,ac52da994cd9c830,0395eb3152abc9f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:20.353 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,119a0c9465eec9f1,302e306696656c32] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=119a0c9465eec9f1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:20.353 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,119a0c9465eec9f1,302e306696656c32] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=119a0c9465eec9f1, duration=1 ms] 2024-10-16 13:24:20.809 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:24:20.936 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,503eacf77e2816ae,10c50af4ec974003] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=503eacf77e2816ae, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:20.937 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,503eacf77e2816ae,10c50af4ec974003] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=503eacf77e2816ae, duration=2 ms] 2024-10-16 13:24:20.959 INFO 4486 --- [ocp-async-4,c92835ec75084297,3335987befd3005c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:20.996 INFO 4486 --- [prometheus-cache-monitor1,1a7199cca6e2ff80,f6331edc7098cc72] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8244} 2024-10-16 13:24:20.996 INFO 4486 --- [prometheus-cache-monitor1,1a7199cca6e2ff80,f6331edc7098cc72] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:24:21.456 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:22.362 INFO 4486 --- [ocp-schedules-7,f6cbcff529fd20b4,617af861f32f7a11] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:24:22.364 INFO 4486 --- [ocp-schedules-7,f6cbcff529fd20b4,617af861f32f7a11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:22.397 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:24:22.398 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.402 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:24:22.410 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.411 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:24:22.414 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:24:22.415 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.416 INFO 4486 --- [ocp-schedules-29,f4c6ed94dc17a9f9,5d4873a756530d1d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:24:22.419 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:24:22.419 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.420 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:24:22.428 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.429 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:24:22.437 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.437 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.438 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:24:22.438 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:24:22.446 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.447 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:22.450 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.450 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:24:22.451 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:24:22.451 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.456 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:24:22.457 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.457 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:22.461 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.462 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:22.465 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.466 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:24:22.467 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.468 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:24:22.473 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.474 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:22.475 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.476 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:24:22.479 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.480 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:24:22.484 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.485 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:24:22.492 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.493 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:24:22.495 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.496 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:24:22.497 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.497 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:24:22.500 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.501 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.501 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.502 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.503 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.503 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.504 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.505 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.505 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.506 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.506 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.507 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.507 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.508 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.508 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:24:22.509 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.509 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.510 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.511 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.512 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.512 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.513 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.514 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.514 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.515 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.516 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.516 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:24:22.517 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.518 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:24:22.519 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.520 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.520 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.520 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:22.523 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:24:22.523 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.524 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.526 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.527 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.531 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.532 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.532 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.533 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:22.535 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.536 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.539 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.540 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.541 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:22.542 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.542 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.542 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:22.543 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:24:22.544 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.545 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.548 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.549 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.552 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.553 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:24:22.558 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:24:22.560 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:24:22.560 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:24:22.575 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.577 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.579 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.580 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.583 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.584 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.587 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.589 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.589 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:22.591 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.591 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.592 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:24:22.592 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.594 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.595 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.598 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.599 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:24:22.602 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.603 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:24:22.605 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.606 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.606 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:24:22.606 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.606 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:24:22.606 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.606 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:24:22.607 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:24:22.609 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:24:22.609 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:24:22.609 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:24:22.611 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:24:22.662 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:22.664 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:22.665 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:24:22.674 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.674 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:24:22.674 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.674 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:24:22.674 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:24:22.674 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:24:22.677 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:24:22.677 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:24:22.959 INFO 4486 --- [ocp-async-7,f3ec4c6bcb0917bd,ecf9e0202de74fa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:22.970 INFO 4486 --- [ocp-async-5,05eb70d3c224eb5e,2db4887e9092317b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:24:22.994 INFO 4486 --- [prometheus-minute-schedule-1,b2015f67b2e6853b,7a8455afe0dd35ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:22.998 INFO 4486 --- [prometheus-minute-schedule-1,b2015f67b2e6853b,7a8455afe0dd35ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:23.762 INFO 4486 --- [ocp-schedules-19,95b30056addcabfe,468cdc5af28b5128] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:24:23.762 INFO 4486 --- [ocp-schedules-19,95b30056addcabfe,468cdc5af28b5128] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:24:23.800 INFO 4486 --- [Collect all tenants major compactions2,95b30056addcabfe,7aa742d228e8871d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:24:23.800 INFO 4486 --- [Collect all tenants major compactions1,95b30056addcabfe,eacafff44c114ff9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:24:23.800 INFO 4486 --- [Collect all tenants major compactions3,95b30056addcabfe,eb37941badad9514] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:24:23.800 INFO 4486 --- [Collect all tenants major compactions4,95b30056addcabfe,7f99d78557eb863a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:24:23.801 INFO 4486 --- [Collect all tenants major compactions5,95b30056addcabfe,0d9fbba7b97e41f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:24:23.812 INFO 4486 --- [Collect all tenants major compactions1,95b30056addcabfe,eacafff44c114ff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:23.813 INFO 4486 --- [Collect all tenants major compactions2,95b30056addcabfe,7aa742d228e8871d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:23.814 INFO 4486 --- [Collect all tenants major compactions1,95b30056addcabfe,eacafff44c114ff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:24:23.815 INFO 4486 --- [Collect all tenants major compactions2,95b30056addcabfe,7aa742d228e8871d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:24:23.817 INFO 4486 --- [Collect all tenants major compactions1,95b30056addcabfe,eacafff44c114ff9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:24:23.818 INFO 4486 --- [Collect all tenants major compactions1,95b30056addcabfe,eacafff44c114ff9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:24:23.818 INFO 4486 --- [Collect all tenants major compactions2,95b30056addcabfe,7aa742d228e8871d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:24:23.818 INFO 4486 --- [Collect all tenants major compactions2,95b30056addcabfe,7aa742d228e8871d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:24:23.820 INFO 4486 --- [Collect all tenants major compactions3,95b30056addcabfe,eb37941badad9514] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:23.823 INFO 4486 --- [Collect all tenants major compactions3,95b30056addcabfe,eb37941badad9514] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:24:23.826 INFO 4486 --- [Collect all tenants major compactions3,95b30056addcabfe,eb37941badad9514] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:24:23.826 INFO 4486 --- [Collect all tenants major compactions3,95b30056addcabfe,eb37941badad9514] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:24:23.830 INFO 4486 --- [Collect all tenants major compactions4,95b30056addcabfe,7f99d78557eb863a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:23.831 INFO 4486 --- [Collect all tenants major compactions5,95b30056addcabfe,0d9fbba7b97e41f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:23.832 INFO 4486 --- [Collect all tenants major compactions4,95b30056addcabfe,7f99d78557eb863a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:24:23.833 INFO 4486 --- [Collect all tenants major compactions5,95b30056addcabfe,0d9fbba7b97e41f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:24:23.836 INFO 4486 --- [Collect all tenants major compactions4,95b30056addcabfe,7f99d78557eb863a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:24:23.836 INFO 4486 --- [Collect all tenants major compactions4,95b30056addcabfe,7f99d78557eb863a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:24:23.838 INFO 4486 --- [Collect all tenants major compactions5,95b30056addcabfe,0d9fbba7b97e41f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:24:23.838 INFO 4486 --- [Collect all tenants major compactions5,95b30056addcabfe,0d9fbba7b97e41f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:24:23.839 INFO 4486 --- [ocp-schedules-19,95b30056addcabfe,468cdc5af28b5128] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:24:23.849 INFO 4486 --- [ocp-schedules-5,3b25a04c95e4526a,353351bfcd5d0596] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:24:24.457 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:24.959 INFO 4486 --- [ocp-async-2,e0d2f40e330e1e34,af2d88ea85681f7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:25.994 INFO 4486 --- [prometheus-minute-schedule-1,db9503dc3e8797f6,9514eda16202b9ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:26.000 INFO 4486 --- [prometheus-minute-schedule-1,db9503dc3e8797f6,9514eda16202b9ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:26.495 INFO 4486 --- [ocp-schedules-25,7852798d011642b4,f106d2452c24199c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:24:26.495+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:24:26.496 INFO 4486 --- [ocp-schedules-25,7852798d011642b4,f106d2452c24199c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:24:26.497 INFO 4486 --- [ocp-schedules-25,7852798d011642b4,f106d2452c24199c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:24:26.498 INFO 4486 --- [ocp-schedules-25,7852798d011642b4,f106d2452c24199c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:24:26.680 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dc02653ca8698627,4c65beb92d11250a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dc02653ca8698627, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:26.680 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dc02653ca8698627,4c65beb92d11250a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dc02653ca8698627, duration=1 ms] 2024-10-16 13:24:26.959 INFO 4486 --- [ocp-async-4,02e88f2833fd704c,3b6b66107da879da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:27.458 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:28.269 INFO 4486 --- [ocp-schedules-23,e6b92eaf3791e8b7,95b94538a85b9286] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:24:28.306 WARN 4486 --- [scanner-13226652-1,e6b92eaf3791e8b7,8a0ba3fe4c685d2c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:24:28.394 INFO 4486 --- [alarm-detect-7,e6b92eaf3791e8b7,f25d59746700b983] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056240 ~ 1729056258 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056255, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056255, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056255=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:24:28.861 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9c925e65c8dd1af4,51128aa4fde11031] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9c925e65c8dd1af4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:28.861 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9c925e65c8dd1af4,51128aa4fde11031] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9c925e65c8dd1af4, duration=2 ms] 2024-10-16 13:24:28.959 INFO 4486 --- [ocp-async-7,b4100356fa5ecdfa,962fc240eeeb535f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:28.994 INFO 4486 --- [prometheus-minute-schedule-1,2daf4a28e5327075,e1ea0b50360e6652] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:29.002 INFO 4486 --- [prometheus-minute-schedule-1,2daf4a28e5327075,e1ea0b50360e6652] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:29.746 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,67254baf7b11552c,3ee334da35d12b87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=67254baf7b11552c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:24:29.746 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,67254baf7b11552c,3ee334da35d12b87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=67254baf7b11552c, duration=1 ms] 2024-10-16 13:24:29.979 INFO 4486 --- [ocp-schedules-20,1397d627b146ea7a,2826e8d89cc234b6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:24:30.103 INFO 4486 --- [prometheus-second-schedule-1,bdd334bc950fddd3,4ca4baf1d00d9ea3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:24:30.357 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e7909c1b66217c6f,e54f3f929dc0b6ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e7909c1b66217c6f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:30.358 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e7909c1b66217c6f,e54f3f929dc0b6ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e7909c1b66217c6f, duration=1 ms] 2024-10-16 13:24:30.458 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:30.941 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e5b36a82ae398a81,856bd43be6a44f8c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e5b36a82ae398a81, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:30.941 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e5b36a82ae398a81,856bd43be6a44f8c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e5b36a82ae398a81, duration=1 ms] 2024-10-16 13:24:30.960 INFO 4486 --- [ocp-async-3,060994c12b9aff01,a93cf35944b8def6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:31.994 INFO 4486 --- [prometheus-minute-schedule-1,2b156f4ac048c61d,849c4744f2e5eb2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:32.000 INFO 4486 --- [prometheus-minute-schedule-1,2b156f4ac048c61d,849c4744f2e5eb2f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:32.960 INFO 4486 --- [ocp-async-8,a31ac033297ed1af,cee2e0cf574ab314] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:32.970 INFO 4486 --- [ocp-async-4,c21c2c827fae6503,6a47dbfc2412aed7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:24:33.459 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:34.960 INFO 4486 --- [ocp-async-6,e237349ca565657c,584cee6cee95026f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:34.994 INFO 4486 --- [prometheus-minute-schedule-1,34505926c6d60fe9,71177ead9fb35597] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:35.000 INFO 4486 --- [prometheus-minute-schedule-1,34505926c6d60fe9,71177ead9fb35597] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:36.460 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:36.695 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c8607701ea268245,4459606cdaff2735] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c8607701ea268245, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:36.696 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c8607701ea268245,4459606cdaff2735] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c8607701ea268245, duration=2 ms] 2024-10-16 13:24:36.960 INFO 4486 --- [ocp-async-3,3c353ea8ef9ad926,57a9f5d83854fc63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:37.063 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:24:37.064 INFO 4486 --- [ocp-schedules-6,097f4d7869a8095b,0f9063d75130c602] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:24:37.066 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:24:37.066 INFO 4486 --- [ocp-schedules-30,e772b58a3f2a2ca0,4af449193f77b57a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:24:37.067 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:24:37.068 INFO 4486 --- [ocp-schedules-30,e772b58a3f2a2ca0,4af449193f77b57a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:37.072 INFO 4486 --- [ocp-schedules-6,097f4d7869a8095b,0f9063d75130c602] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:24:37.081 INFO 4486 --- [ocp-schedules-8,e73192046060a1e6,8a955740e1192c2d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:24:37.083 INFO 4486 --- [ocp-schedules-8,e73192046060a1e6,8a955740e1192c2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:24:37.090 INFO 4486 --- [ocp-schedules-1,10faebb14e7a43b7,3e335ac4136c4514] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:24:37.091 INFO 4486 --- [ocp-schedules-13,d9d6171b118fa6a3,2de614f815cc30b3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:24:37.091 INFO 4486 --- [ocp-schedules-5,c6c6b00bccd86abb,eb19d536ed1f155a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:24:37.091 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:24:37.091 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:24:37.091 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:24:37.091 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:24:37.092 INFO 4486 --- [host-check-agent-status1,c6c6b00bccd86abb,6a21c4abda85e4ea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:24:37.092 INFO 4486 --- [host-check-agent-status2,c6c6b00bccd86abb,1bb71b1515706c34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:24:37.092 INFO 4486 --- [host-check-agent-status3,c6c6b00bccd86abb,9d679b5f7888718f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:24:37.093 INFO 4486 --- [host-check-agent-status4,c6c6b00bccd86abb,1a5f5a2bfb983783] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:24:37.094 INFO 4486 --- [host-check-clockdiff2,d9d6171b118fa6a3,9298334dbc8b0b08] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:24:37.095 INFO 4486 --- [host-check-clockdiff1,d9d6171b118fa6a3,2d9cc909d2e0fe8f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:24:37.095 INFO 4486 --- [host-check-clockdiff3,d9d6171b118fa6a3,71dd3b48d3c8ba57] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:24:37.096 INFO 4486 --- [host-check-clockdiff4,d9d6171b118fa6a3,05508320bb066711] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:24:37.101 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:37.101 INFO 4486 --- [host-check-agent-status1,c6c6b00bccd86abb,6a21c4abda85e4ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:37.101 INFO 4486 --- [host-check-agent-status3,c6c6b00bccd86abb,9d679b5f7888718f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:37.109 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:37.109 INFO 4486 --- [host-check-agent-status2,c6c6b00bccd86abb,1bb71b1515706c34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:37.109 INFO 4486 --- [host-check-agent-status4,c6c6b00bccd86abb,1a5f5a2bfb983783] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:37.109 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:37.109 INFO 4486 --- [host-check-agent-status1,c6c6b00bccd86abb,6a21c4abda85e4ea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:37.110 INFO 4486 --- [host-check-agent-status3,c6c6b00bccd86abb,9d679b5f7888718f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:37.110 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:37.118 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:37.124 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:37.124 INFO 4486 --- [host-check-clockdiff2,d9d6171b118fa6a3,9298334dbc8b0b08] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056277 3 3 , error= 2024-10-16 13:24:37.122 INFO 4486 --- [host-check-agent-status4,c6c6b00bccd86abb,1a5f5a2bfb983783] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:37.133 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:37.133 INFO 4486 --- [host-check-agent-status2,c6c6b00bccd86abb,1bb71b1515706c34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:37.140 INFO 4486 --- [host-check-clockdiff1,d9d6171b118fa6a3,2d9cc909d2e0fe8f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056277 0 0 , error= 2024-10-16 13:24:37.141 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:37.133 INFO 4486 --- [host-check-clockdiff4,d9d6171b118fa6a3,05508320bb066711] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056277 3 3 , error= 2024-10-16 13:24:37.146 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.147 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:37.148 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:24:37.165 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.166 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:24:37.166 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:24:37.166 INFO 4486 --- [host-check-agent-status2,c6c6b00bccd86abb,1bb71b1515706c34] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:24:37.166 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:24:37.166 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:37.166 INFO 4486 --- [host-check-agent-status4,c6c6b00bccd86abb,1a5f5a2bfb983783] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:24:37.167 INFO 4486 --- [host-check-agent-status1,c6c6b00bccd86abb,6a21c4abda85e4ea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:24:37.171 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:24:37.172 INFO 4486 --- [host-check-agent-status3,c6c6b00bccd86abb,9d679b5f7888718f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:24:37.172 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:24:37.174281550+08:00 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:24:37.175133920+08:00 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status2,c6c6b00bccd86abb,1bb71b1515706c34] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status2,c6c6b00bccd86abb,1bb71b1515706c34] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status1,c6c6b00bccd86abb,6a21c4abda85e4ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status2,c6c6b00bccd86abb,1bb71b1515706c34] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status1,c6c6b00bccd86abb,6a21c4abda85e4ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status1,c6c6b00bccd86abb,6a21c4abda85e4ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:24:37.175+08:00 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:24:37.175+08:00 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status4,c6c6b00bccd86abb,1a5f5a2bfb983783] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:24:37.175584875+08:00 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status4,c6c6b00bccd86abb,1a5f5a2bfb983783] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:24:37.175 INFO 4486 --- [host-check-agent-status4,c6c6b00bccd86abb,1a5f5a2bfb983783] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:24:37.176 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:24:37.178531671+08:00 2024-10-16 13:24:37.176 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:24:37.175+08:00 2024-10-16 13:24:37.176 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:24:37.176+08:00 2024-10-16 13:24:37.178 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:24:37.175+08:00 and status AVAILABLE 2024-10-16 13:24:37.178 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:24:37.175+08:00 and status AVAILABLE 2024-10-16 13:24:37.178 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:24:37.175+08:00 and status AVAILABLE 2024-10-16 13:24:37.179 INFO 4486 --- [host-check-agent-status3,c6c6b00bccd86abb,9d679b5f7888718f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:24:37.179 INFO 4486 --- [host-check-agent-status3,c6c6b00bccd86abb,9d679b5f7888718f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:24:37.179 INFO 4486 --- [host-check-agent-status3,c6c6b00bccd86abb,9d679b5f7888718f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:24:37.179 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:24:37.176+08:00 and status AVAILABLE 2024-10-16 13:24:37.180 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:24:37.180 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:24:37.181 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:24:37.183 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.183 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:24:37.184 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.186 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:24:37.187 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:24:37.187 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.188 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.188 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:37.188 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:37.189 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:37.189 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:24:37.190 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:37.190 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:37.190 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:37.192 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:37.194 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:37.196 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.197 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:37.200 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.200 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.200 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.200 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.200 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.201 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.207 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.209 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:37.215 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:37.215 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:37.217 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:37.218 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:24:37.222 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.223 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:24:37.225 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.226 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:24:37.230 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:24:37.236 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.238 INFO 4486 --- [ocp-schedules-25,166ef2a1ff3f2b34,fa82020b4ab425f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:24:37.247 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:24:37.253 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:24:37.253 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:24:37.259 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:24:37.267 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:24:37.267 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:24:37.280 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:37.286 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:37.288 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:37.292 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:24:37.304 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:24:37.313 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:24:37.323 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:24:37.325 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:24:37.329 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:24:37.330 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:24:37.341 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:24:37.342 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:24:37.343 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:37.361 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:37.362 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:37.376 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:24:37.380 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:24:37.387 INFO 4486 --- [host-check-host-status3,10faebb14e7a43b7,d2b17ad22704a7d0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:24:37.392 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:24:37.396 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:24:37.399 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:24:37.407 INFO 4486 --- [host-check-host-status4,10faebb14e7a43b7,05abfe1ba9a1ad78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:24:37.409 INFO 4486 --- [host-check-host-status2,10faebb14e7a43b7,3a1eac0476e18999] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:24:37.411 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:24:37.463 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:37.465 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.466 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:24:37.467 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.467 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.469 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:24:37.470 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:24:37.480 INFO 4486 --- [host-check-host-status1,10faebb14e7a43b7,44bc4fa8b2d5250b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:24:37.480 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.480 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.480 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.480 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.480 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.481 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.509 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:24:37.547 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:24:37.592 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:37.607 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:37.608 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.609 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:24:37.610 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:24:37.611 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.611 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:24:37.612 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.613 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.614 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:24:37.622 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.622 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.622 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.622 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.622 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.623 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.630 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:24:37.861 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:24:37.909 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:37.911 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.912 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:24:37.913 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.913 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.914 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:24:37.924 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.924 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.924 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.924 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.924 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:37.925 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:37.943 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:24:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:24:38.012 INFO 4486 --- [prometheus-minute-schedule-1,f674ee8117cfe176,99a86564df9610c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:38.013 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:24:38.014 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7f1912b6b5d63fdb,384df238d86e2e25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=7f1912b6b5d63fdb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:24:38.015 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7f1912b6b5d63fdb,384df238d86e2e25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=7f1912b6b5d63fdb, duration=2 ms] 2024-10-16 13:24:38.020 INFO 4486 --- [prometheus-minute-schedule-1,f674ee8117cfe176,99a86564df9610c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:38.076 INFO 4486 --- [host-check-clockdiff3,d9d6171b118fa6a3,71dd3b48d3c8ba57] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056278 3 3 , error= 2024-10-16 13:24:38.218 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:24:38.263 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:38.265 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:38.266 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:24:38.267 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:38.268 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:38.269 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:24:38.279 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:38.279 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:38.279 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:38.279 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:38.279 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:24:38.280 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:38.355 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:24:38.465 INFO 4486 --- [ocp-schedules-20,02312ab4278d11fb,45b0f572290b9066] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:24:38.466 INFO 4486 --- [ocp-schedules-20,02312ab4278d11fb,45b0f572290b9066] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:38.475 INFO 4486 --- [ocp-schedules-5,1a153722c66f10ac,5ee9661936222ba2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:24:38.513 WARN 4486 --- [scanner-13226920-1,1a153722c66f10ac,b1b736ae87afd121] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:24:38.540 INFO 4486 --- [alarm-detect-6,1a153722c66f10ac,3133c52104b3a5e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:38.544 INFO 4486 --- [alarm-detect-6,1a153722c66f10ac,3133c52104b3a5e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:38.549 INFO 4486 --- [alarm-detect-6,1a153722c66f10ac,3133c52104b3a5e1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:38.551 INFO 4486 --- [alarm-detect-19,1a153722c66f10ac,567084cfe3644c44] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:38.552 INFO 4486 --- [alarm-detect-2,1a153722c66f10ac,a656fdf2b641568d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:38.555 INFO 4486 --- [alarm-detect-19,1a153722c66f10ac,567084cfe3644c44] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:38.557 INFO 4486 --- [alarm-detect-2,1a153722c66f10ac,a656fdf2b641568d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:38.558 INFO 4486 --- [alarm-detect-19,1a153722c66f10ac,567084cfe3644c44] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:38.561 INFO 4486 --- [alarm-detect-2,1a153722c66f10ac,a656fdf2b641568d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:38.605 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:24:38.358624, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:24:38.606 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:24:38.358624, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:24:38.630 INFO 4486 --- [alarm-detect-2,1a153722c66f10ac,3b3f7adce8015db2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:38.637 INFO 4486 --- [alarm-detect-2,1a153722c66f10ac,3b3f7adce8015db2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:38.642 INFO 4486 --- [alarm-detect-2,1a153722c66f10ac,3b3f7adce8015db2] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:38.653 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:24:38.358624} } 2024-10-16 13:24:38.713 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:24:38.719 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:24:38.721 INFO 4486 --- [ocp-schedules-10,602e8c85699a5e61,783190219e180195] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:24:38.728 INFO 4486 --- [ocp-schedules-26,415812a34feb5af3,a80dc3acdd69047d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:24:38.865 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dd3daa69e8a5bb99,e22de87df26d498f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dd3daa69e8a5bb99, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:38.865 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dd3daa69e8a5bb99,e22de87df26d498f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dd3daa69e8a5bb99, duration=1 ms] 2024-10-16 13:24:38.961 INFO 4486 --- [ocp-async-10,7fadce28049ca4c7,932472d823c30da1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:39.461 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:39.985 INFO 4486 --- [ocp-schedules-22,602512789fb45322,7aea3e6da77e2df0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:24:40.007 INFO 4486 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:24:40.007 INFO 4486 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:24:40.027 INFO 4486 --- [Check obproxy2,2a53a88d35bbc8c5,9679ba3045c6dd6d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:24:40.027 INFO 4486 --- [Check obproxy1,4cea33ded2d96cb1,f7768f7b0dc8cc3e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:24:40.027 INFO 4486 --- [Check obproxy2,2a53a88d35bbc8c5,9679ba3045c6dd6d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:40.027 INFO 4486 --- [Check obproxy1,4cea33ded2d96cb1,f7768f7b0dc8cc3e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:40.034 INFO 4486 --- [Collect obproxy1,d9c196f5bd339e0d,1458472ee214f23e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:24:40.034 INFO 4486 --- [Collect obproxy1,d9c196f5bd339e0d,1458472ee214f23e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:40.034 INFO 4486 --- [Collect obproxy2,257666566dc21858,f1f284a49547abb6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:24:40.034 INFO 4486 --- [Collect obproxy2,257666566dc21858,f1f284a49547abb6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:24:40.048 INFO 4486 --- [Check obproxy2,2a53a88d35bbc8c5,9679ba3045c6dd6d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:24:40.048 INFO 4486 --- [Check obproxy2,2a53a88d35bbc8c5,9679ba3045c6dd6d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:24:40.056 INFO 4486 --- [Check obproxy1,4cea33ded2d96cb1,f7768f7b0dc8cc3e] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:24:40.056 INFO 4486 --- [Collect obproxy2,257666566dc21858,f1f284a49547abb6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:24:40.056 INFO 4486 --- [Check obproxy1,4cea33ded2d96cb1,f7768f7b0dc8cc3e] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:24:40.059 INFO 4486 --- [Collect obproxy1,d9c196f5bd339e0d,1458472ee214f23e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:24:40.061 INFO 4486 --- [Collect obproxy2,257666566dc21858,f1f284a49547abb6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:24:40.065 INFO 4486 --- [Collect obproxy1,d9c196f5bd339e0d,1458472ee214f23e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:24:40.067 INFO 4486 --- [Collect obproxy2,257666566dc21858,f1f284a49547abb6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:24:40.069 INFO 4486 --- [Collect obproxy1,d9c196f5bd339e0d,1458472ee214f23e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:24:40.362 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e28f867789f9cc17,f3fabb0b0c370151] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e28f867789f9cc17, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:40.362 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e28f867789f9cc17,f3fabb0b0c370151] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e28f867789f9cc17, duration=1 ms] 2024-10-16 13:24:40.946 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6e83fe436763a597,d4dfc20ab433c380] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6e83fe436763a597, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:40.946 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6e83fe436763a597,d4dfc20ab433c380] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6e83fe436763a597, duration=1 ms] 2024-10-16 13:24:40.961 INFO 4486 --- [ocp-async-5,28fa92dde08a517b,2e866f57b10a5ce8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:40.994 INFO 4486 --- [prometheus-minute-schedule-1,948613e86ab24dfa,7a827cc2f0a2e4d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:40.995 INFO 4486 --- [prometheus-hour-schedule-1,18869f47ca20dac4,c561b71fadc8dc1c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:24:40.999 INFO 4486 --- [prometheus-minute-schedule-1,948613e86ab24dfa,7a827cc2f0a2e4d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:41.649 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 13:24:41.649 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 13:24:41.650 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.651 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:24:41.652 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.653 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.654 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.654 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 13:24:41.673 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.673 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.674 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.675 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 13:24:41.690 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.690 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.691 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.691 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 13:24:41.706 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.706 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.707 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.707 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 13:24:41.721 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.721 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.722 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.722 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 13:24:41.730 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.730 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.731 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.731 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 13:24:41.737 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.737 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.738 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.739 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 13:24:41.750 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.751 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.752 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.752 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 13:24:41.760 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.760 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.761 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.762 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 13:24:41.769 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.769 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.770 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.770 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 13:24:41.777 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.777 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.778 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.779 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 13:24:41.788 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.789 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.789 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.790 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 13:24:41.806 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.807 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.807 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.808 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 13:24:41.815 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.816 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.817 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.818 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 13:24:41.826 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.827 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.828 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.828 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 13:24:41.840 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.841 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.841 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.842 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 13:24:41.848 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.849 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.850 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.851 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 13:24:41.856 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.856 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.857 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.857 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 13:24:41.862 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.863 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.863 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.864 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 13:24:41.874 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.874 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.875 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.876 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 13:24:41.882 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.883 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.884 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.885 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 13:24:41.890 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.891 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.892 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.893 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 13:24:41.897 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.898 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.899 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.899 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 13:24:41.904 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.904 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.905 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.906 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 13:24:41.911 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.912 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.913 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.913 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 13:24:41.917 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.918 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.919 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.920 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 13:24:41.925 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.925 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.926 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.926 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 13:24:41.931 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.932 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.933 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.933 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 13:24:41.938 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.939 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.939 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.940 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 13:24:41.945 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.946 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.947 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.947 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 13:24:41.952 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.953 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.954 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.954 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 13:24:41.959 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.959 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.960 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.960 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 13:24:41.965 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.966 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.966 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.967 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 13:24:41.973 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.973 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.974 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.974 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 13:24:41.982 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.983 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.984 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.984 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 13:24:41.991 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.991 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:41.992 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.993 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 13:24:41.999 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:41.999 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.000 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.000 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 13:24:42.006 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.007 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.008 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.008 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 13:24:42.013 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.014 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.015 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.015 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 13:24:42.019 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.020 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.021 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.021 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 13:24:42.025 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.026 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.027 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.027 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 13:24:42.032 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.032 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.033 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.033 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 13:24:42.043 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.044 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.045 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.045 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 13:24:42.053 INFO 4486 --- [ocp-schedules-13,9766f30695e571c7,df57df1b2692b7b0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:24:42.066 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.067 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.067 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.068 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 13:24:42.075 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.075 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.076 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.076 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 13:24:42.083 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.084 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.085 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.086 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 13:24:42.093 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.093 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.094 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.095 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 13:24:42.099 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.099 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.100 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.101 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 13:24:42.106 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.106 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.107 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.108 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 13:24:42.112 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 13:24:42.112 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 13:24:42.120 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.120 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:24:42.122 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.122 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.123 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.125 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 13:24:42.132 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.132 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:24:42.134 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:42.134 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 13:24:42.139 INFO 4486 --- [ocp-schedules-17,956daa826eb8a563,cc2274c6bd0a2097] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 13:24:42.462 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:42.961 INFO 4486 --- [ocp-async-2,f819cee083d142e4,e508de7e744e89b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:42.971 INFO 4486 --- [ocp-async-1,593ff78268eec6e0,51cf544d4c107bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:24:43.994 INFO 4486 --- [prometheus-minute-schedule-1,9eaf8c6b850cde5f,1343db628b716f68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:44.001 INFO 4486 --- [prometheus-minute-schedule-1,9eaf8c6b850cde5f,1343db628b716f68] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:44.961 INFO 4486 --- [ocp-async-10,5336103e0a24ec83,0c2eebf3c81d5ea3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:45.391 INFO 4486 --- [ocp-schedules-21,38eae43e7cb6dc63,1cdcd4a93472e273] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:24:45.398 INFO 4486 --- [ocp-schedules-21,38eae43e7cb6dc63,1cdcd4a93472e273] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:24:45.398 INFO 4486 --- [ocp-schedules-21,38eae43e7cb6dc63,1cdcd4a93472e273] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:24:45.406 INFO 4486 --- [ocp-schedules-29,e3ff1fe0430c0429,78042bf55ead3150] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:24:45.406 INFO 4486 --- [ocp-schedules-29,e3ff1fe0430c0429,78042bf55ead3150] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:24:45.463 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:46.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ae7e075f5c9ac1a6,e658784cc294120e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ae7e075f5c9ac1a6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:46.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ae7e075f5c9ac1a6,e658784cc294120e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ae7e075f5c9ac1a6, duration=2 ms] 2024-10-16 13:24:46.961 INFO 4486 --- [ocp-async-6,880dbe2233783bb8,e6a912aea1890970] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:46.994 INFO 4486 --- [prometheus-minute-schedule-1,af2dd8c1cebe54aa,63abc1d117682bb9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:47.000 INFO 4486 --- [prometheus-minute-schedule-1,af2dd8c1cebe54aa,63abc1d117682bb9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:47.027 INFO 4486 --- [ocp-schedules-24,fb284115dce52228,2892c520f2c06623] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:24:47.070 INFO 4486 --- [ocp-schedules-30,e772b58a3f2a2ca0,4af449193f77b57a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:24:47.172 INFO 4486 --- [ocp-schedules-8,e73192046060a1e6,8a955740e1192c2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:24:48.016 INFO 4486 --- [ocp-schedules-16,d475b2fd4edf4180,cd01f1aa4f95ba44] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:24:48.017 INFO 4486 --- [ocp-schedules-16,c24adbabc6615db6,232639b1584650ef] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:24:48.019 INFO 4486 --- [ocp-schedules-16,c24adbabc6615db6,232639b1584650ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:24:48.046 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.047 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:48.059 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.060 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:48.077 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.078 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.078 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:24:48.081 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.082 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:48.104 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.105 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.105 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:48.106 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:24:48.106 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.109 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:24:48.110 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:48.120 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.120 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:48.129 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.138 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.142 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:48.143 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:24:48.149 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:24:48.154 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:48.156 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:48.180 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:24:48.185 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.186 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.188 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:24:48.227 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:24:48.227 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:24:48.238 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.240 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.241 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:24:48.245 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.248 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:24:48.251 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:48.254 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:24:48.265 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.266 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.267 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:24:48.276 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:24:48.276 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:24:48.278 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.278 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:48.304 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.304 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:48.330 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.330 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:48.355 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.355 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:24:48.381 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.381 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:24:48.406 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:24:48.435 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.436 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.437 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:24:48.442 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:24:48.446 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:24:48.452 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:24:48.453 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:24:48.464 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:48.477 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:24:48.502 INFO 4486 --- [ocp-schedules-20,02312ab4278d11fb,45b0f572290b9066] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:24:48.626 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:24:48.653 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:24:48.654 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:24:48.655 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:24:48.675 INFO 4486 --- [ocp-schedules-18,4dad242134bea16b,9a806a83a2b60e95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:24:48.685 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:24:48.685 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:24:48.721 WARN 4486 --- [scanner-13227236-1,4dad242134bea16b,beaa68ba6560cb3f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:24:48.765 INFO 4486 --- [alarm-detect-2,4dad242134bea16b,1e4f30ae8ac943fe] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056255 ~ 1729056278 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056270, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056270, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056270=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:24:48.869 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b6b889dddd2d4e2c,24f0147211ceb013] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b6b889dddd2d4e2c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:48.869 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b6b889dddd2d4e2c,24f0147211ceb013] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b6b889dddd2d4e2c, duration=1 ms] 2024-10-16 13:24:48.961 INFO 4486 --- [ocp-async-2,feb9c5aa17ff1bfc,20f25ba961682f06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:49.375 INFO 4486 --- [ocp-schedules-14,5279df894ba69de4,137ed39d41f4b5e2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:24:49.376 INFO 4486 --- [ocp-schedules-14,5279df894ba69de4,137ed39d41f4b5e2] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:24:49.376 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:24:49.376 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:24:49.376 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:24:49.376 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:24:49.376 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:24:49.381 INFO 4486 --- [ocp-schedules-14,5279df894ba69de4,137ed39d41f4b5e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:24:49.381 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:24:49.381 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:24:49.381 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:24:49.382 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:24:49.382 INFO 4486 --- [ocp-schedules-14,5279df894ba69de4,137ed39d41f4b5e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:24:49.383 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:24:49.383 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:24:49.383 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:24:49.383 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:24:49.384 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:24:49.384 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:24:49.406 INFO 4486 --- [ocp-schedules-14,5279df894ba69de4,137ed39d41f4b5e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:24:49.407 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:24:49.407 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:24:49.408 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:24:49.410 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:24:49.411 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:24:49.413 INFO 4486 --- [ocp-schedules-14,5279df894ba69de4,137ed39d41f4b5e2] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:24:49.991 INFO 4486 --- [ocp-schedules-22,18b03df334e46490,12546023fc6db9db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:24:49.994 INFO 4486 --- [prometheus-minute-schedule-1,7411f3e1d65b8538,d441fde9e2aed375] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:49.999 INFO 4486 --- [prometheus-minute-schedule-1,7411f3e1d65b8538,d441fde9e2aed375] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:50.312 INFO 4486 --- [ocp-schedules-9,e8a552d88ba63bdf,3281eba78c16cf29] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:24:50.312 INFO 4486 --- [ocp-schedules-9,e8a552d88ba63bdf,3281eba78c16cf29] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:24:50.313 INFO 4486 --- [ocp-schedules-9,e8a552d88ba63bdf,3281eba78c16cf29] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:24:50.313 INFO 4486 --- [ocp-schedules-9,e8a552d88ba63bdf,3281eba78c16cf29] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:24:50.314 INFO 4486 --- [ocp-schedules-9,e8a552d88ba63bdf,3281eba78c16cf29] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:24:50.314 INFO 4486 --- [ocp-schedules-9,e8a552d88ba63bdf,3281eba78c16cf29] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703033600770641920 2024-10-16 13:24:50.366 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,30754bf2d08cc11f,8c2acc414ef3eb25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=30754bf2d08cc11f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:50.366 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,30754bf2d08cc11f,8c2acc414ef3eb25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=30754bf2d08cc11f, duration=1 ms] 2024-10-16 13:24:50.472 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:24:50.472 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:24:50.484 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:24:50.484 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:24:50.484 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:24:50.494 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:24:50.494 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:24:50.494 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:24:50.502 INFO 4486 --- [ocp-schedules-5,d472e81db87dadab,0b93653bdaaf8b0a] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:24:50.952 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,64264027acce063c,a1e5465d244ddafe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=64264027acce063c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:50.952 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,64264027acce063c,a1e5465d244ddafe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=64264027acce063c, duration=1 ms] 2024-10-16 13:24:50.962 INFO 4486 --- [ocp-async-4,0ea14eda4667941c,d841a0531004f8c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:51.465 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:52.962 INFO 4486 --- [ocp-async-7,c9c303ed02479b92,c6997443cbfc4c74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:52.971 INFO 4486 --- [ocp-async-5,45058e68d258a142,0f1c8fa55d4c43ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:24:52.994 INFO 4486 --- [prometheus-minute-schedule-1,ae559a0f10bb8f52,85b897d364c47151] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:53.000 INFO 4486 --- [prometheus-minute-schedule-1,ae559a0f10bb8f52,85b897d364c47151] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:54.466 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:54.962 INFO 4486 --- [ocp-async-2,635258bb096a58e0,23cb6b6b6aa1d482] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:55.994 INFO 4486 --- [prometheus-minute-schedule-1,8e9aeb0efd3b448d,60005e2373611275] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:56.001 INFO 4486 --- [prometheus-minute-schedule-1,8e9aeb0efd3b448d,60005e2373611275] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:56.707 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a0d4cc8d25e9792b,2554524a5d915971] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a0d4cc8d25e9792b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:56.707 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a0d4cc8d25e9792b,2554524a5d915971] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a0d4cc8d25e9792b, duration=2 ms] 2024-10-16 13:24:56.962 INFO 4486 --- [ocp-async-4,9418b88e3e969d55,400426a0f3f42fff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:57.467 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:24:58.732 INFO 4486 --- [ocp-schedules-24,52706f1ee4f85d2c,80b094196e4125a0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:24:58.841 INFO 4486 --- [ocp-schedules-9,03aa2fa8cd94c0b7,2cca2bcf4e6952c8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:24:58.876 WARN 4486 --- [scanner-13227455-1,03aa2fa8cd94c0b7,3110c190a4e21689] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:24:58.886 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8188083bcea086e3,4945fae3d97c2da8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8188083bcea086e3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:24:58.886 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8188083bcea086e3,4945fae3d97c2da8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8188083bcea086e3, duration=11 ms] 2024-10-16 13:24:58.903 INFO 4486 --- [alarm-detect-8,03aa2fa8cd94c0b7,39d0f21707798aff] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:58.907 INFO 4486 --- [alarm-detect-8,03aa2fa8cd94c0b7,39d0f21707798aff] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:58.911 INFO 4486 --- [alarm-detect-8,03aa2fa8cd94c0b7,39d0f21707798aff] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:24:58.923 INFO 4486 --- [alarm-detect-20,03aa2fa8cd94c0b7,1e895456fcac5f23] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:58.926 INFO 4486 --- [alarm-detect-6,03aa2fa8cd94c0b7,cc86824fe3e26d76] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:58.927 INFO 4486 --- [alarm-detect-20,03aa2fa8cd94c0b7,1e895456fcac5f23] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:58.929 INFO 4486 --- [alarm-detect-6,03aa2fa8cd94c0b7,cc86824fe3e26d76] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:58.930 INFO 4486 --- [alarm-detect-20,03aa2fa8cd94c0b7,1e895456fcac5f23] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:24:58.933 INFO 4486 --- [alarm-detect-6,03aa2fa8cd94c0b7,cc86824fe3e26d76] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:24:58.949 INFO 4486 --- [alarm-detect-9,03aa2fa8cd94c0b7,c4a18dbb20c07ad0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056270 ~ 1729056288 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056285, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056285, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056285=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:24:58.963 INFO 4486 --- [ocp-async-7,d38826d86fe0ec40,36c1a01ea4311564] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:24:58.994 INFO 4486 --- [prometheus-minute-schedule-1,472f40bde1ce8e93,2c674d5dc44bbe39] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056300, interval=60 2024-10-16 13:24:59.013 INFO 4486 --- [prometheus-minute-schedule-1,472f40bde1ce8e93,2c674d5dc44bbe39] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:24:59.015 INFO 4486 --- [alarm-detect-17,03aa2fa8cd94c0b7,64ea650984244c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:59.021 INFO 4486 --- [alarm-detect-17,03aa2fa8cd94c0b7,64ea650984244c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:59.028 INFO 4486 --- [alarm-detect-17,03aa2fa8cd94c0b7,64ea650984244c3e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:24:59.996 INFO 4486 --- [ocp-schedules-28,c351c6cf51f63ac0,b387fef0d12550d1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:25:00.318 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,8ba11d828e451a05,abbcf4d7f8e1ba7a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=8ba11d828e451a05, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:25:00.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,8ba11d828e451a05,abbcf4d7f8e1ba7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=8ba11d828e451a05, duration=2 ms] 2024-10-16 13:25:00.333 INFO 4486 --- [ocp-schedules-13,d1e2c88149392951,daa2acf226d6d163] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:25:00.333+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:25:00.338 INFO 4486 --- [ocp-schedules-13,d1e2c88149392951,daa2acf226d6d163] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:25:00.354 INFO 4486 --- [ocp-schedules-13,d1e2c88149392951,daa2acf226d6d163] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:25:00.368 INFO 4486 --- [ocp-schedules-13,d1e2c88149392951,daa2acf226d6d163] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:25:00.370 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,542fd6a4adba64de,e6646dc730d10521] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=542fd6a4adba64de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:00.370 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,542fd6a4adba64de,e6646dc730d10521] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=542fd6a4adba64de, duration=1 ms] 2024-10-16 13:25:00.403 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:20:01.0, nextRun=2024-10-16 13:25:00.0, currentDate=Wed Oct 16 13:25:00 CST 2024 2024-10-16 13:25:00.403 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:20:01.0, nextRunTime=2024-10-16 13:25:00.0} 2024-10-16 13:25:00.407 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-10-16 13:25:00.407 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.197.49.184', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-10-16T13:25:00.407+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='791e68b70ade5cad', operation=EXECUTE} 2024-10-16 13:25:00.424 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:20:01.0, nextRun=2024-10-16 13:25:00.0, currentDate=Wed Oct 16 13:25:00 CST 2024 2024-10-16 13:25:00.424 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:20:01.0, nextRunTime=2024-10-16 13:25:00.0} 2024-10-16 13:25:00.454 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 1000003] 2024-10-16 13:25:00.468 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:00.474 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.p.s.internal.SqlParseTaskSchedule : The last sql parse service is still running, skip current task 2024-10-16 13:25:00.474 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,2001bb484e076cc0] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-10-16 13:25:00.956 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ed22d4eb212dffd8,cc383a4549c64b94] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ed22d4eb212dffd8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:00.957 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ed22d4eb212dffd8,cc383a4549c64b94] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ed22d4eb212dffd8, duration=1 ms] 2024-10-16 13:25:00.963 INFO 4486 --- [ocp-async-3,28b9cd6545595c01,7c96f5d1b3e71c4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:01.067 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,3636274a297bde59] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7931961, operation:EXECUTE, state: RUNNING 2024-10-16 13:25:01.085 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7931961, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266486, task_operation=execute, latest_execution_start_time=2024-10-16T13:25:01.059+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=7931961}, listMap={}}, executor=10.197.49.184 2024-10-16 13:25:01.117 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.178 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.354 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.395 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.412 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for OceanBase 2024-10-16 13:25:01.412 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1079.4750314957316], b:[-1.850753682719265E12] 2024-10-16 13:25:01.417 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.488 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.529 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.559 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:25:01.631 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:01.668 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:01.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,33895fef0c8bbdd5,e40d31c716acef66] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=33895fef0c8bbdd5, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 13:25:01.926 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:01.951 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,33895fef0c8bbdd5,e40d31c716acef66] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=33895fef0c8bbdd5, duration=249 ms] 2024-10-16 13:25:01.993 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:01.994 INFO 4486 --- [prometheus-minute-schedule-1,a24670b399993f83,789068abc6979fb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:01.999 INFO 4486 --- [prometheus-minute-schedule-1,a24670b399993f83,789068abc6979fb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:02.042 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for obce_3zones 2024-10-16 13:25:02.042 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1295.9606763372929], b:[2.2870925145727866E12] 2024-10-16 13:25:02.046 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:02.194 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:02.227 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:02.253 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:25:02.401 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.436 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.648 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.685 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.707 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for glob_3zone 2024-10-16 13:25:02.707 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1292.3504806553328], b:[2.281317553551766E12] 2024-10-16 13:25:02.711 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.847 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.877 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.904 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:25:02.963 INFO 4486 --- [ocp-async-8,9de057f4b223b355,a08c6af82349e61f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:02.971 INFO 4486 --- [ocp-async-4,08275a3966e43a2e,d02b7ec63debc812] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:25:03.057 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,a71cf9f518ab8eca] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7931961, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 13:25:03.425 INFO 4486 --- [ocp-schedules-18,a7206efff695894a,c2e409b4dd871888] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:25:03.427 INFO 4486 --- [ocp-schedules-18,a7206efff695894a,c2e409b4dd871888] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:25:03.428 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:25:03.428 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:25:03.429 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:25:03.469 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:04.963 INFO 4486 --- [ocp-async-7,6234db45da4706ab,59329199a58feded] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:04.994 INFO 4486 --- [prometheus-minute-schedule-1,d68c0f59b5bc353c,17ec0335c5da7d2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:05.000 INFO 4486 --- [prometheus-minute-schedule-1,d68c0f59b5bc353c,17ec0335c5da7d2b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:06.469 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:06.711 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1125e93d6d0625b8,4434056c91027844] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1125e93d6d0625b8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:06.712 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1125e93d6d0625b8,4434056c91027844] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1125e93d6d0625b8, duration=2 ms] 2024-10-16 13:25:06.964 INFO 4486 --- [ocp-async-3,65cedbb5167cc074,8ac8f873df2aacf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:07.066 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:25:07.066 INFO 4486 --- [ocp-schedules-23,4657de7c74e64f83,8bd45013278bdea7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:25:07.068 INFO 4486 --- [ocp-schedules-23,4657de7c74e64f83,8bd45013278bdea7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:25:07.081 INFO 4486 --- [ocp-schedules-14,94e6f2defef7005c,fb63aec305cb7196] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:25:07.083 INFO 4486 --- [ocp-schedules-14,94e6f2defef7005c,fb63aec305cb7196] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:25:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:25:07.090 INFO 4486 --- [ocp-schedules-3,c60e33201ecdac8e,0677a83aa07c8bdd] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:25:07.090 INFO 4486 --- [ocp-schedules-9,eb93a16e385a54d9,771c9d5fb6e40cc6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:25:07.090 INFO 4486 --- [ocp-schedules-2,f9df3255f17bbf87,b1f24db39b25faf6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:25:07.090 INFO 4486 --- [host-check-clockdiff1,c60e33201ecdac8e,42e2e609e68c1d64] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:25:07.091 INFO 4486 --- [host-check-clockdiff2,c60e33201ecdac8e,f05a3331e95a669c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:25:07.091 INFO 4486 --- [host-check-clockdiff3,c60e33201ecdac8e,6d7b1848968b73d6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:25:07.091 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:25:07.091 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:25:07.091 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:25:07.092 INFO 4486 --- [host-check-agent-status1,eb93a16e385a54d9,6b35abb3e7d1c802] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:25:07.093 INFO 4486 --- [host-check-clockdiff4,c60e33201ecdac8e,aea563105f69b300] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:25:07.093 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:25:07.094 INFO 4486 --- [host-check-agent-status2,eb93a16e385a54d9,41b998783dbccac0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:25:07.094 INFO 4486 --- [host-check-agent-status3,eb93a16e385a54d9,a4c216815cb7f10b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:25:07.095 INFO 4486 --- [host-check-agent-status4,eb93a16e385a54d9,202fb023dd489b98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:25:07.099 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:25:07.100 INFO 4486 --- [host-check-agent-status1,eb93a16e385a54d9,6b35abb3e7d1c802] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:07.115 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:07.115 INFO 4486 --- [host-check-agent-status2,eb93a16e385a54d9,41b998783dbccac0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:25:07.101 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:25:07.115 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:25:07.115 INFO 4486 --- [host-check-agent-status1,eb93a16e385a54d9,6b35abb3e7d1c802] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:07.129 INFO 4486 --- [host-check-clockdiff4,c60e33201ecdac8e,aea563105f69b300] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056307 2 2 , error= 2024-10-16 13:25:07.129 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:25:07.130 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:07.132 INFO 4486 --- [host-check-clockdiff1,c60e33201ecdac8e,42e2e609e68c1d64] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056307 0 0 , error= 2024-10-16 13:25:07.132 INFO 4486 --- [host-check-agent-status4,eb93a16e385a54d9,202fb023dd489b98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:25:07.132 INFO 4486 --- [host-check-agent-status2,eb93a16e385a54d9,41b998783dbccac0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:25:07.132 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:25:07.132 INFO 4486 --- [host-check-agent-status3,eb93a16e385a54d9,a4c216815cb7f10b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:25:07.134 INFO 4486 --- [host-check-clockdiff2,c60e33201ecdac8e,f05a3331e95a669c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056307 3 3 , error= 2024-10-16 13:25:07.135 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:07.135 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:25:07.136 INFO 4486 --- [host-check-agent-status4,eb93a16e385a54d9,202fb023dd489b98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:25:07.136 INFO 4486 --- [host-check-agent-status3,eb93a16e385a54d9,a4c216815cb7f10b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:25:07.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:07.143 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:25:07.156 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:07.158 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.159 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:25:07.159 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:25:07.160 INFO 4486 --- [host-check-agent-status2,eb93a16e385a54d9,41b998783dbccac0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:25:07.160 INFO 4486 --- [host-check-agent-status3,eb93a16e385a54d9,a4c216815cb7f10b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:25:07.160 INFO 4486 --- [host-check-agent-status1,eb93a16e385a54d9,6b35abb3e7d1c802] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:25:07.160 INFO 4486 --- [host-check-agent-status4,eb93a16e385a54d9,202fb023dd489b98] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:25:07.161 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:25:07.161 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:25:07.164 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.165 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:07.165 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:25:07.165 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:25:07.164606217+08:00 2024-10-16 13:25:07.165 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:25:07.168394918+08:00 2024-10-16 13:25:07.166 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:25:07.165+08:00 2024-10-16 13:25:07.166 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:25:07.169107894+08:00 2024-10-16 13:25:07.166 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:25:07.166+08:00 2024-10-16 13:25:07.166 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:25:07.166+08:00 2024-10-16 13:25:07.167 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:25:07.169464098+08:00 2024-10-16 13:25:07.167 INFO 4486 --- [host-check-agent-status3,eb93a16e385a54d9,a4c216815cb7f10b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:25:07.167 INFO 4486 --- [host-check-agent-status3,eb93a16e385a54d9,a4c216815cb7f10b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:25:07.167 INFO 4486 --- [host-check-agent-status3,eb93a16e385a54d9,a4c216815cb7f10b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:25:07.168 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:25:07.167+08:00 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status1,eb93a16e385a54d9,6b35abb3e7d1c802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status2,eb93a16e385a54d9,41b998783dbccac0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status2,eb93a16e385a54d9,41b998783dbccac0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status2,eb93a16e385a54d9,41b998783dbccac0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status1,eb93a16e385a54d9,6b35abb3e7d1c802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status1,eb93a16e385a54d9,6b35abb3e7d1c802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:25:07.165+08:00 and status AVAILABLE 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:25:07.166+08:00 and status AVAILABLE 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:25:07.166+08:00 and status AVAILABLE 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status4,eb93a16e385a54d9,202fb023dd489b98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status4,eb93a16e385a54d9,202fb023dd489b98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:25:07.169 INFO 4486 --- [host-check-agent-status4,eb93a16e385a54d9,202fb023dd489b98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:25:07.170 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:25:07.167+08:00 and status AVAILABLE 2024-10-16 13:25:07.171 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:25:07.171 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:25:07.172 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:25:07.172 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:25:07.172 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.173 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:25:07.173 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:25:07.180 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:07.180 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:25:07.180 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:25:07.181 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:25:07.181 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:25:07.181 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:07.181 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:25:07.182 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:25:07.183 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.184 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:25:07.193 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.194 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:25:07.196 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.197 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:25:07.204 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:07.204 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:07.204 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:07.205 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:07.209 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:07.210 INFO 4486 --- [ocp-schedules-24,1e074da389f8c77b,28dff091b7f665d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:25:07.233 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:25:07.239 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:25:07.246 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:25:07.250 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:25:07.252 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:25:07.258 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:25:07.264 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:07.265 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:25:07.270 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:25:07.270 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:07.279 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:07.287 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:07.293 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:25:07.303 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:25:07.313 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:25:07.319 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:07.321 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:25:07.324 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:25:07.331 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:25:07.341 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:07.343 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:25:07.346 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:25:07.350 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:07.357 INFO 4486 --- [host-check-host-status3,f9df3255f17bbf87,30fbef3c63304721] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:25:07.361 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:25:07.374 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:25:07.380 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:07.385 INFO 4486 --- [host-check-host-status4,f9df3255f17bbf87,891c90f7eb723f25] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:25:07.386 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:25:07.395 INFO 4486 --- [host-check-host-status2,f9df3255f17bbf87,bc36d19846df967b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:25:07.431 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:25:07.441 INFO 4486 --- [host-check-host-status1,f9df3255f17bbf87,359ec1d4f74db1a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:25:07.994 INFO 4486 --- [prometheus-minute-schedule-1,6d3a4b635dd5f77e,0411a88cd243f0c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:08.001 INFO 4486 --- [prometheus-minute-schedule-1,6d3a4b635dd5f77e,0411a88cd243f0c8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:08.058 INFO 4486 --- [host-check-clockdiff3,c60e33201ecdac8e,6d7b1848968b73d6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056308 3 3 , error= 2024-10-16 13:25:08.893 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5d5ec5e0cee1d475,6bd81f1c23823c76] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5d5ec5e0cee1d475, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:08.893 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5d5ec5e0cee1d475,6bd81f1c23823c76] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5d5ec5e0cee1d475, duration=1 ms] 2024-10-16 13:25:08.964 INFO 4486 --- [ocp-async-8,e297d32f935f7369,6e0f99afea936be7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:09.099 INFO 4486 --- [ocp-schedules-19,9954991bc3b687fc,5f6689e6c4939fc5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:25:09.139 WARN 4486 --- [scanner-13228086-1,9954991bc3b687fc,9b5975870e64ba38] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:25:09.470 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:09.640 INFO 4486 --- [ocp-schedules-26,c50fb0dc95414c69,1d5e908165cb3b0d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 13:25:09.640 INFO 4486 --- [ocp-schedules-26,c50fb0dc95414c69,1d5e908165cb3b0d] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 13:25:09.673 INFO 4486 --- [pool-276-thread-1,c50fb0dc95414c69,e13be7dfa915d52e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:25:09.673 INFO 4486 --- [pool-276-thread-2,c50fb0dc95414c69,4dd07c523b3d5069] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:25:09.673 INFO 4486 --- [pool-276-thread-3,c50fb0dc95414c69,4ad1fb926165f6fe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:25:09.693 INFO 4486 --- [pool-276-thread-1,c50fb0dc95414c69,e13be7dfa915d52e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:09.694 INFO 4486 --- [pool-276-thread-2,c50fb0dc95414c69,4dd07c523b3d5069] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:09.696 INFO 4486 --- [pool-276-thread-1,c50fb0dc95414c69,e13be7dfa915d52e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056309000000] 2024-10-16 13:25:09.702 INFO 4486 --- [pool-276-thread-2,c50fb0dc95414c69,4dd07c523b3d5069] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056309000000] 2024-10-16 13:25:09.706 INFO 4486 --- [pool-276-thread-3,c50fb0dc95414c69,4ad1fb926165f6fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:09.714 INFO 4486 --- [pool-276-thread-3,c50fb0dc95414c69,4ad1fb926165f6fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056309000000] 2024-10-16 13:25:09.999 WARN 4486 --- [pool-276-thread-3,c50fb0dc95414c69,4ad1fb926165f6fe] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 13:25:09.999 WARN 4486 --- [pool-276-thread-3,c50fb0dc95414c69,4ad1fb926165f6fe] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 13:25:10.001 INFO 4486 --- [ocp-schedules-4,1a944c5714724860,33fa68715ccc8c84] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:25:10.001 INFO 4486 --- [pool-276-thread-3,c50fb0dc95414c69,4ad1fb926165f6fe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:25:10.006 INFO 4486 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:25:10.008 INFO 4486 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:25:10.025 INFO 4486 --- [Check obproxy2,c6aa6b5f914b93ff,c8b5c1191c7b5888] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:25:10.025 INFO 4486 --- [Check obproxy2,c6aa6b5f914b93ff,c8b5c1191c7b5888] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:10.027 INFO 4486 --- [Check obproxy1,031428c65b97e238,73112cfc93c40c0a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:25:10.027 INFO 4486 --- [Check obproxy1,031428c65b97e238,73112cfc93c40c0a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:10.030 INFO 4486 --- [Collect obproxy1,76ba3dd4681f8ab2,8849d4e76be22642] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:25:10.030 INFO 4486 --- [Collect obproxy1,76ba3dd4681f8ab2,8849d4e76be22642] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:10.033 INFO 4486 --- [Collect obproxy2,12d2f01b3605262c,040e5976097cb79d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:25:10.033 INFO 4486 --- [Collect obproxy2,12d2f01b3605262c,040e5976097cb79d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:10.047 INFO 4486 --- [Check obproxy2,c6aa6b5f914b93ff,c8b5c1191c7b5888] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:25:10.047 INFO 4486 --- [Check obproxy2,c6aa6b5f914b93ff,c8b5c1191c7b5888] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:25:10.048 INFO 4486 --- [Check obproxy1,031428c65b97e238,73112cfc93c40c0a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:25:10.048 INFO 4486 --- [Check obproxy1,031428c65b97e238,73112cfc93c40c0a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:25:10.057 INFO 4486 --- [Collect obproxy2,12d2f01b3605262c,040e5976097cb79d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:25:10.060 INFO 4486 --- [Collect obproxy1,76ba3dd4681f8ab2,8849d4e76be22642] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:25:10.063 INFO 4486 --- [Collect obproxy2,12d2f01b3605262c,040e5976097cb79d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:25:10.065 INFO 4486 --- [Collect obproxy1,76ba3dd4681f8ab2,8849d4e76be22642] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:25:10.067 INFO 4486 --- [Collect obproxy2,12d2f01b3605262c,040e5976097cb79d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:25:10.069 INFO 4486 --- [Collect obproxy1,76ba3dd4681f8ab2,8849d4e76be22642] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:25:10.315 WARN 4486 --- [pool-276-thread-1,c50fb0dc95414c69,e13be7dfa915d52e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 13:25:10.317 INFO 4486 --- [pool-276-thread-1,c50fb0dc95414c69,e13be7dfa915d52e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:25:10.375 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,aae3358f8a077555,6edd93afbbce6df6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=aae3358f8a077555, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:10.375 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,aae3358f8a077555,6edd93afbbce6df6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=aae3358f8a077555, duration=1 ms] 2024-10-16 13:25:10.756 INFO 4486 --- [pool-276-thread-2,c50fb0dc95414c69,4dd07c523b3d5069] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1053, sql=SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id; 2024-10-16 13:25:10.756 WARN 4486 --- [pool-276-thread-2,c50fb0dc95414c69,4dd07c523b3d5069] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 13:25:10.758 INFO 4486 --- [pool-276-thread-2,c50fb0dc95414c69,4dd07c523b3d5069] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:25:10.761 INFO 4486 --- [ocp-schedules-26,c50fb0dc95414c69,1d5e908165cb3b0d] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 13:25:10.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a785ae7d3d66f45c,56636b435f0ed0d1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a785ae7d3d66f45c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:10.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a785ae7d3d66f45c,56636b435f0ed0d1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a785ae7d3d66f45c, duration=1 ms] 2024-10-16 13:25:10.964 INFO 4486 --- [ocp-async-6,b9e07207a4f887d2,b5b4ac048b966164] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:25:10.994 INFO 4486 --- [prometheus-minute-schedule-1,fc04d408d1c39c57,4323038624553a9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:11.000 INFO 4486 --- [prometheus-minute-schedule-1,fc04d408d1c39c57,4323038624553a9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:11.008 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:25:11.133 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,97948baa6ac90589,a44d90844eff71a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=97948baa6ac90589, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:25:11.134 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,97948baa6ac90589,a44d90844eff71a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=97948baa6ac90589, duration=2 ms] 2024-10-16 13:25:12.471 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:12.830 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056240, nodeCount=860, affectRows=860 2024-10-16 13:25:12.833 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056240, nodeCount=878, affectRows=878 2024-10-16 13:25:12.835 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056240, nodeCount=860, validCount=860 2024-10-16 13:25:12.836 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056240, nodeCount=877, validCount=877 2024-10-16 13:25:12.836 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056240, nodeCount=827, affectRows=827 2024-10-16 13:25:12.839 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056240, nodeCount=827, validCount=827 2024-10-16 13:25:12.841 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056240, nodeCount=885, affectRows=885 2024-10-16 13:25:12.843 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056240, nodeCount=883, validCount=883 2024-10-16 13:25:12.964 INFO 4486 --- [ocp-async-9,68f05e0c9761088b,396fe519dbc309a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:12.971 INFO 4486 --- [ocp-async-3,1e68b8f748a554aa,baaa9ff768199fb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:25:13.429 INFO 4486 --- [ocp-schedules-18,a7206efff695894a,c2e409b4dd871888] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:25:13.994 INFO 4486 --- [prometheus-minute-schedule-1,b5543139419bb3a0,d4d34ac2d93c8646] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:14.002 INFO 4486 --- [prometheus-minute-schedule-1,b5543139419bb3a0,d4d34ac2d93c8646] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:14.213 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056240, nodeCount=18, affectRows=18 2024-10-16 13:25:14.216 INFO 4486 --- [metric-archive-async-16,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056240, nodeCount=61, affectRows=61 2024-10-16 13:25:14.217 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056240, nodeCount=77, affectRows=77 2024-10-16 13:25:14.218 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056240, nodeCount=90, affectRows=90 2024-10-16 13:25:14.220 INFO 4486 --- [metric-archive-async-10,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056240, nodeCount=128, affectRows=128 2024-10-16 13:25:14.234 INFO 4486 --- [metric-archive-async-11,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056240, nodeCount=106, affectRows=106 2024-10-16 13:25:14.248 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056240, nodeCount=18, validCount=18 2024-10-16 13:25:14.316 INFO 4486 --- [metric-archive-async-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056240, nodeCount=61, validCount=61 2024-10-16 13:25:14.354 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056240, nodeCount=77, validCount=77 2024-10-16 13:25:14.362 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056240, nodeCount=90, validCount=90 2024-10-16 13:25:14.391 INFO 4486 --- [metric-archive-async-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056240, nodeCount=106, validCount=106 2024-10-16 13:25:14.418 INFO 4486 --- [metric-archive-async-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056240, nodeCount=128, validCount=128 2024-10-16 13:25:14.964 INFO 4486 --- [ocp-async-1,2e1f1515007b82b2,0ff53e6525268e53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:15.473 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:16.716 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3b1f684cf3b31dd1,54a8a9f9825cffd4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3b1f684cf3b31dd1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:16.716 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3b1f684cf3b31dd1,54a8a9f9825cffd4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3b1f684cf3b31dd1, duration=1 ms] 2024-10-16 13:25:16.965 INFO 4486 --- [ocp-async-4,77afefca7cf142f3,a99ecddf775ccfc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:16.994 INFO 4486 --- [prometheus-minute-schedule-1,82d5dee3360562ae,47bf03d808b76417] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:17.000 INFO 4486 --- [prometheus-minute-schedule-1,82d5dee3360562ae,47bf03d808b76417] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:17.030 INFO 4486 --- [ocp-schedules-27,e499023a791a7951,eaf27b0b808f70b2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:25:17.070 INFO 4486 --- [ocp-schedules-23,4657de7c74e64f83,8bd45013278bdea7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:25:17.172 INFO 4486 --- [ocp-schedules-14,94e6f2defef7005c,fb63aec305cb7196] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:25:18.473 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:18.686 INFO 4486 --- [ocp-schedules-16,c24adbabc6615db6,232639b1584650ef] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:25:18.686 INFO 4486 --- [ocp-schedules-16,95ac45397782ebdf,d6ef498246c64d0c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:25:18.688 INFO 4486 --- [ocp-schedules-16,95ac45397782ebdf,d6ef498246c64d0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:25:18.717 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.718 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:18.730 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.731 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:18.736 INFO 4486 --- [ocp-schedules-1,e6f7cc7240f8ebe9,3f14a3d286e41fdc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:25:18.752 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.753 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:18.761 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:18.762 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.763 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:25:18.805 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:18.806 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.806 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:18.808 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:25:18.808 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.811 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:25:18.813 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:18.827 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:18.827 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:18.883 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:18.893 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:18.894 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.894 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:18.895 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:25:18.897 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,192485b0141aa363,2a34142147c99974] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=192485b0141aa363, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:18.897 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,192485b0141aa363,2a34142147c99974] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=192485b0141aa363, duration=1 ms] 2024-10-16 13:25:18.900 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:25:18.906 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:18.908 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:18.928 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:18.929 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.930 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:25:18.937 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:25:18.965 INFO 4486 --- [ocp-async-5,94e02190b47c58fd,2e4269b9131b6a4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:18.983 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:18.984 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.985 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:18.986 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:25:18.986 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:25:18.988 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:18.992 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:25:18.995 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:19.013 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:25:19.022 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:19.022 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:19.022 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:19.023 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:19.024 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:25:19.033 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:25:19.033 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:25:19.051 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:19.051 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:19.079 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:19.079 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:19.107 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:19.107 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:19.135 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:19.135 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:19.164 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:19.190 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:19.192 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:19.193 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:19.197 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:25:19.201 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:25:19.206 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:25:19.208 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:25:19.217 INFO 4486 --- [ocp-schedules-9,a3293f8f1f0dd2a8,ea3d88d3d8ce693b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:25:19.237 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:25:19.267 WARN 4486 --- [scanner-13229160-1,a3293f8f1f0dd2a8,b6a843e5a3782415] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:25:19.283 INFO 4486 --- [alarm-detect-11,a3293f8f1f0dd2a8,1c609beef7048409] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:25:19.289 INFO 4486 --- [alarm-detect-11,a3293f8f1f0dd2a8,1c609beef7048409] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:25:19.296 INFO 4486 --- [alarm-detect-11,a3293f8f1f0dd2a8,1c609beef7048409] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:25:19.315 INFO 4486 --- [alarm-detect-1,a3293f8f1f0dd2a8,7d24e1f23eb33c2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:25:19.321 INFO 4486 --- [alarm-detect-11,a3293f8f1f0dd2a8,d5665cc74f5fc681] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:25:19.322 INFO 4486 --- [alarm-detect-1,a3293f8f1f0dd2a8,7d24e1f23eb33c2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:25:19.326 INFO 4486 --- [alarm-detect-11,a3293f8f1f0dd2a8,d5665cc74f5fc681] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:25:19.327 INFO 4486 --- [alarm-detect-1,a3293f8f1f0dd2a8,7d24e1f23eb33c2e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:25:19.331 INFO 4486 --- [alarm-detect-11,a3293f8f1f0dd2a8,d5665cc74f5fc681] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:25:19.382 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:25:19.383 INFO 4486 --- [alarm-detect-9,a3293f8f1f0dd2a8,9bc13616d15bca8c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056285 ~ 1729056309 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056300, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056300, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056300=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:25:19.388 INFO 4486 --- [alarm-detect-9,a3293f8f1f0dd2a8,9bc13616d15bca8c] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:25:19.417 INFO 4486 --- [alarm-detect-6,a3293f8f1f0dd2a8,48aba0b4fb4774f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:25:19.418 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:19.419 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:19.420 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:25:19.422 INFO 4486 --- [alarm-detect-6,a3293f8f1f0dd2a8,48aba0b4fb4774f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:25:19.427 INFO 4486 --- [alarm-detect-6,a3293f8f1f0dd2a8,48aba0b4fb4774f9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:25:19.460 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:25:19.460 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:25:19.994 INFO 4486 --- [prometheus-minute-schedule-1,7991a4c205fe1be9,baf25a022e56bfdf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:20.000 INFO 4486 --- [prometheus-minute-schedule-1,7991a4c205fe1be9,baf25a022e56bfdf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:20.006 INFO 4486 --- [ocp-schedules-4,c41ec21d35291a47,cfc131b0397b978e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:25:20.381 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,914b20b6c27fd01e,c1585985d8459d08] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=914b20b6c27fd01e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:20.381 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,914b20b6c27fd01e,c1585985d8459d08] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=914b20b6c27fd01e, duration=1 ms] 2024-10-16 13:25:20.965 INFO 4486 --- [ocp-async-3,3f2481086d2d45a3,71d71cb0e945b747] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:20.966 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,da357bd6a2d9e2b8,ab00a8426c0f7329] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=da357bd6a2d9e2b8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:20.966 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,da357bd6a2d9e2b8,ab00a8426c0f7329] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=da357bd6a2d9e2b8, duration=1 ms] 2024-10-16 13:25:20.996 INFO 4486 --- [prometheus-cache-monitor1,cb8d64eebceb1911,21095e8ef0814045] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8244} 2024-10-16 13:25:20.996 INFO 4486 --- [prometheus-cache-monitor1,cb8d64eebceb1911,21095e8ef0814045] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:25:21.474 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:22.448 INFO 4486 --- [ocp-schedules-2,16ec7be7dfc0d306,6ddfe0e7b50f0f32] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:25:22.678 INFO 4486 --- [ocp-schedules-7,f6cbcff529fd20b4,617af861f32f7a11] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:25:22.965 INFO 4486 --- [ocp-async-1,588ef9049b9fa4e1,65216f48f793d5fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:22.972 INFO 4486 --- [ocp-async-6,9d3b05928efffef3,8a242576a42254ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:25:22.994 INFO 4486 --- [prometheus-minute-schedule-1,385f0ca03ae89d87,01f18a255f593ec4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:23.003 INFO 4486 --- [prometheus-minute-schedule-1,385f0ca03ae89d87,01f18a255f593ec4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:23.839 INFO 4486 --- [ocp-schedules-9,a4d7ff8f35dd5e2d,1a7d74d0861ea790] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:25:23.839 INFO 4486 --- [ocp-schedules-9,a4d7ff8f35dd5e2d,1a7d74d0861ea790] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:25:23.878 INFO 4486 --- [Collect all tenants major compactions2,a4d7ff8f35dd5e2d,05c212f8cf00c529] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:25:23.878 INFO 4486 --- [Collect all tenants major compactions4,a4d7ff8f35dd5e2d,a037dda50173f921] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:25:23.878 INFO 4486 --- [Collect all tenants major compactions5,a4d7ff8f35dd5e2d,0da79824b22a9ebf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:25:23.878 INFO 4486 --- [Collect all tenants major compactions3,a4d7ff8f35dd5e2d,2fcd254ab5b5a306] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:25:23.879 INFO 4486 --- [Collect all tenants major compactions1,a4d7ff8f35dd5e2d,b0423854385c9da3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:25:23.885 INFO 4486 --- [ocp-schedules-12,866c233e438300f1,f822bf482f47afa3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:25:23.891 INFO 4486 --- [Collect all tenants major compactions2,a4d7ff8f35dd5e2d,05c212f8cf00c529] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:23.892 INFO 4486 --- [Collect all tenants major compactions2,a4d7ff8f35dd5e2d,05c212f8cf00c529] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:25:23.892 INFO 4486 --- [Collect all tenants major compactions1,a4d7ff8f35dd5e2d,b0423854385c9da3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:23.894 INFO 4486 --- [Collect all tenants major compactions1,a4d7ff8f35dd5e2d,b0423854385c9da3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:25:23.896 INFO 4486 --- [Collect all tenants major compactions3,a4d7ff8f35dd5e2d,2fcd254ab5b5a306] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:23.897 INFO 4486 --- [Collect all tenants major compactions2,a4d7ff8f35dd5e2d,05c212f8cf00c529] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:25:23.898 INFO 4486 --- [Collect all tenants major compactions2,a4d7ff8f35dd5e2d,05c212f8cf00c529] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:25:23.898 INFO 4486 --- [Collect all tenants major compactions1,a4d7ff8f35dd5e2d,b0423854385c9da3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:25:23.898 INFO 4486 --- [Collect all tenants major compactions1,a4d7ff8f35dd5e2d,b0423854385c9da3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:25:23.898 INFO 4486 --- [Collect all tenants major compactions3,a4d7ff8f35dd5e2d,2fcd254ab5b5a306] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:25:23.902 INFO 4486 --- [Collect all tenants major compactions3,a4d7ff8f35dd5e2d,2fcd254ab5b5a306] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:25:23.902 INFO 4486 --- [Collect all tenants major compactions3,a4d7ff8f35dd5e2d,2fcd254ab5b5a306] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:25:23.912 INFO 4486 --- [Collect all tenants major compactions4,a4d7ff8f35dd5e2d,a037dda50173f921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:23.912 INFO 4486 --- [Collect all tenants major compactions5,a4d7ff8f35dd5e2d,0da79824b22a9ebf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:23.913 INFO 4486 --- [Collect all tenants major compactions4,a4d7ff8f35dd5e2d,a037dda50173f921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:25:23.914 INFO 4486 --- [Collect all tenants major compactions5,a4d7ff8f35dd5e2d,0da79824b22a9ebf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:25:23.917 INFO 4486 --- [Collect all tenants major compactions4,a4d7ff8f35dd5e2d,a037dda50173f921] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:25:23.917 INFO 4486 --- [Collect all tenants major compactions4,a4d7ff8f35dd5e2d,a037dda50173f921] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:25:23.919 INFO 4486 --- [Collect all tenants major compactions5,a4d7ff8f35dd5e2d,0da79824b22a9ebf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:25:23.919 INFO 4486 --- [Collect all tenants major compactions5,a4d7ff8f35dd5e2d,0da79824b22a9ebf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:25:23.921 INFO 4486 --- [ocp-schedules-9,a4d7ff8f35dd5e2d,1a7d74d0861ea790] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:25:24.475 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:24.966 INFO 4486 --- [ocp-async-9,44c9af175c11d6c7,e1864bbf9492fd41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:25.994 INFO 4486 --- [prometheus-minute-schedule-1,aa115d4c989bb85b,0da295b371267699] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:26.000 INFO 4486 --- [prometheus-minute-schedule-1,aa115d4c989bb85b,0da295b371267699] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:26.529 INFO 4486 --- [ocp-schedules-27,3678834f2912f36a,a874937303109b4c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:25:26.529+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:25:26.531 INFO 4486 --- [ocp-schedules-27,3678834f2912f36a,a874937303109b4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:25:26.533 INFO 4486 --- [ocp-schedules-27,3678834f2912f36a,a874937303109b4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:25:26.534 INFO 4486 --- [ocp-schedules-27,3678834f2912f36a,a874937303109b4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:25:26.727 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e99cdbcf2a75a9e7,c57c9d4f5e93a385] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e99cdbcf2a75a9e7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:26.727 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e99cdbcf2a75a9e7,c57c9d4f5e93a385] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e99cdbcf2a75a9e7, duration=2 ms] 2024-10-16 13:25:26.966 INFO 4486 --- [ocp-async-3,d9639864f91ee12e,b51031170a8d2558] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:27.476 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:28.902 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,56527fe0803771c1,0fd35044be83634e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=56527fe0803771c1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:28.902 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,56527fe0803771c1,0fd35044be83634e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=56527fe0803771c1, duration=1 ms] 2024-10-16 13:25:28.966 INFO 4486 --- [ocp-async-10,a66ffd23d786b43e,da538c5eaf623e7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:28.994 INFO 4486 --- [prometheus-minute-schedule-1,276041d2c4c72598,8e12dc4097038849] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:29.000 INFO 4486 --- [prometheus-minute-schedule-1,276041d2c4c72598,8e12dc4097038849] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:29.463 INFO 4486 --- [ocp-schedules-27,91bf5ad5d436d54e,f8563f5f2e85276b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:25:29.494 WARN 4486 --- [scanner-13229494-1,91bf5ad5d436d54e,0be8d12ae85a8b5c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:25:29.555 INFO 4486 --- [alarm-detect-9,91bf5ad5d436d54e,833a8ff0f0a86859] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056300 ~ 1729056319 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056315, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056315, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056315=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:25:30.014 INFO 4486 --- [ocp-schedules-24,82797d70c9ffcb1f,2468a04bf617d3b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:25:30.103 INFO 4486 --- [prometheus-second-schedule-1,2a4d9ea1bf9e1fc3,86277a3e3c1b8d07] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:25:30.387 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,835155a81ce59bf6,0a00490fe6f6313d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=835155a81ce59bf6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:30.388 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,835155a81ce59bf6,0a00490fe6f6313d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=835155a81ce59bf6, duration=2 ms] 2024-10-16 13:25:30.477 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:30.966 INFO 4486 --- [ocp-async-4,75787f94b6e5be59,fd454ea543e53481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:30.971 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,afc3ac596e210a97,b03bea26dddb2212] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=afc3ac596e210a97, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:30.972 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,afc3ac596e210a97,b03bea26dddb2212] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=afc3ac596e210a97, duration=2 ms] 2024-10-16 13:25:31.994 INFO 4486 --- [prometheus-minute-schedule-1,6486048da81fc9b0,48ca725f3b8c92d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:32.003 INFO 4486 --- [prometheus-minute-schedule-1,6486048da81fc9b0,48ca725f3b8c92d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:32.967 INFO 4486 --- [ocp-async-9,0352f6a4e08c9adf,67cd247574ab06f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:32.972 INFO 4486 --- [ocp-async-3,936f277b621df9f1,95f8d90088c3e334] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:25:33.478 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:34.967 INFO 4486 --- [ocp-async-1,368bd699b1c0d0e6,137be0cc539472ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:34.994 INFO 4486 --- [prometheus-minute-schedule-1,a88758628859dfcc,d4a3c22b9f3cde01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:35.006 INFO 4486 --- [prometheus-minute-schedule-1,a88758628859dfcc,d4a3c22b9f3cde01] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:35.178 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cc5752dc4ed26261,2b64f941d265e382] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=cc5752dc4ed26261, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:25:35.179 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cc5752dc4ed26261,2b64f941d265e382] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=cc5752dc4ed26261, duration=3 ms] 2024-10-16 13:25:36.479 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:36.739 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,518161098022ce25,609b246517e3ee9f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=518161098022ce25, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:36.739 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,518161098022ce25,609b246517e3ee9f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=518161098022ce25, duration=1 ms] 2024-10-16 13:25:36.967 INFO 4486 --- [ocp-async-4,d1937105f829aeba,4afb3f7bd6bb35ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:37.063 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:25:37.064 INFO 4486 --- [ocp-schedules-26,828126bbde35f0e2,ffd91065440944b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:25:37.066 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:25:37.066 INFO 4486 --- [ocp-schedules-11,71fb4d5bf32c19dc,f46696ef14293f34] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:25:37.068 INFO 4486 --- [ocp-schedules-11,71fb4d5bf32c19dc,f46696ef14293f34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:25:37.069 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:25:37.071 INFO 4486 --- [ocp-schedules-26,828126bbde35f0e2,ffd91065440944b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:25:37.081 INFO 4486 --- [ocp-schedules-22,bef9fb8f3f8cceeb,0d010c10c84fb55d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:25:37.083 INFO 4486 --- [ocp-schedules-22,bef9fb8f3f8cceeb,0d010c10c84fb55d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:25:37.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:25:37.090 INFO 4486 --- [ocp-schedules-9,a6406282d64b8928,6ab90900018229ac] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:25:37.090 INFO 4486 --- [ocp-schedules-6,35e942a6cf1b120c,fc2d685c7355e69d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:25:37.091 INFO 4486 --- [ocp-schedules-13,550e09997df4fe8f,07bfb2d1c83f275b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:25:37.092 INFO 4486 --- [host-check-agent-status1,550e09997df4fe8f,35910de31207abad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:25:37.092 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:25:37.092 INFO 4486 --- [host-check-clockdiff3,a6406282d64b8928,de36b8ced0a26cff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:25:37.091 INFO 4486 --- [host-check-clockdiff2,a6406282d64b8928,80116d85697fb8fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:25:37.092 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:25:37.091 INFO 4486 --- [host-check-clockdiff1,a6406282d64b8928,57b380e51a1cb7b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:25:37.092 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:25:37.093 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:25:37.094 INFO 4486 --- [host-check-clockdiff4,a6406282d64b8928,818acddc72f6d848] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:25:37.094 INFO 4486 --- [host-check-agent-status3,550e09997df4fe8f,3382911220272668] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:25:37.094 INFO 4486 --- [host-check-agent-status4,550e09997df4fe8f,e3f2afb3974d6ae7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:25:37.094 INFO 4486 --- [host-check-agent-status2,550e09997df4fe8f,590d9582a1c8ad1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:25:37.099 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:37.099 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:25:37.108 INFO 4486 --- [host-check-agent-status1,550e09997df4fe8f,35910de31207abad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:37.124 INFO 4486 --- [host-check-agent-status2,550e09997df4fe8f,590d9582a1c8ad1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:25:37.124 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:25:37.124 INFO 4486 --- [host-check-agent-status4,550e09997df4fe8f,e3f2afb3974d6ae7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:25:37.124 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:25:37.127 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:37.127 INFO 4486 --- [host-check-agent-status1,550e09997df4fe8f,35910de31207abad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:37.127 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:25:37.139 INFO 4486 --- [host-check-agent-status3,550e09997df4fe8f,3382911220272668] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:25:37.139 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:25:37.139 INFO 4486 --- [host-check-agent-status2,550e09997df4fe8f,590d9582a1c8ad1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:25:37.139 INFO 4486 --- [host-check-clockdiff2,a6406282d64b8928,80116d85697fb8fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056337 3 3 , error= 2024-10-16 13:25:37.139 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:25:37.139 INFO 4486 --- [host-check-agent-status4,550e09997df4fe8f,e3f2afb3974d6ae7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:25:37.152 INFO 4486 --- [host-check-clockdiff1,a6406282d64b8928,57b380e51a1cb7b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056337 0 0 , error= 2024-10-16 13:25:37.153 INFO 4486 --- [host-check-agent-status3,550e09997df4fe8f,3382911220272668] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:25:37.154 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.155 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:37.155 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:25:37.175 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.176 INFO 4486 --- [host-check-agent-status4,550e09997df4fe8f,e3f2afb3974d6ae7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:25:37.176 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:37.177 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:25:37.177 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:25:37.178 INFO 4486 --- [host-check-agent-status1,550e09997df4fe8f,35910de31207abad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:25:37.179 INFO 4486 --- [host-check-agent-status3,550e09997df4fe8f,3382911220272668] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:25:37.179 INFO 4486 --- [host-check-agent-status2,550e09997df4fe8f,590d9582a1c8ad1e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:25:37.180 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:25:37.181 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:25:37.183711531+08:00 2024-10-16 13:25:37.181 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:25:37.181 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:25:37.183624241+08:00 2024-10-16 13:25:37.181 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:25:37.182 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:25:37.181+08:00 2024-10-16 13:25:37.182 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:25:37.181+08:00 2024-10-16 13:25:37.182 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:25:37.182229751+08:00 2024-10-16 13:25:37.183 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:25:37.182+08:00 2024-10-16 13:25:37.183 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.183 INFO 4486 --- [host-check-agent-status3,550e09997df4fe8f,3382911220272668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:25:37.183 INFO 4486 --- [host-check-agent-status3,550e09997df4fe8f,3382911220272668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:25:37.183 INFO 4486 --- [host-check-agent-status3,550e09997df4fe8f,3382911220272668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:25:37.186616828+08:00 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-agent-status4,550e09997df4fe8f,e3f2afb3974d6ae7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-agent-status4,550e09997df4fe8f,e3f2afb3974d6ae7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-agent-status4,550e09997df4fe8f,e3f2afb3974d6ae7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-agent-status1,550e09997df4fe8f,35910de31207abad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-agent-status1,550e09997df4fe8f,35910de31207abad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-agent-status1,550e09997df4fe8f,35910de31207abad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:25:37.184 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:25:37.184 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:25:37.184+08:00 2024-10-16 13:25:37.185 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:25:37.181+08:00 and status AVAILABLE 2024-10-16 13:25:37.185 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:25:37.181+08:00 and status AVAILABLE 2024-10-16 13:25:37.185 INFO 4486 --- [host-check-agent-status2,550e09997df4fe8f,590d9582a1c8ad1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:25:37.185 INFO 4486 --- [host-check-agent-status2,550e09997df4fe8f,590d9582a1c8ad1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:25:37.185 INFO 4486 --- [host-check-agent-status2,550e09997df4fe8f,590d9582a1c8ad1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:25:37.185 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:25:37.182+08:00 and status AVAILABLE 2024-10-16 13:25:37.186 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.186 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.187 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:25:37.187 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:25:37.187 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:25:37.187 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:25:37.188 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:25:37.184+08:00 and status AVAILABLE 2024-10-16 13:25:37.189 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.190 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:25:37.195 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:25:37.195 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:25:37.196 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.196 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.196 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.196 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.196 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.196 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:25:37.196 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:37.197 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.197 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:25:37.198 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:25:37.198 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:37.198 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:25:37.200 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:25:37.204 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.205 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:25:37.215 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:25:37.215 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.216 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:25:37.219 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:37.223 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:37.226 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:37.229 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:25:37.230 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.231 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:25:37.233 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.234 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:25:37.244 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.245 INFO 4486 --- [ocp-schedules-12,4723d8b696b8248a,e3b5baba3763f7b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:25:37.251 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:25:37.259 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:25:37.264 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:25:37.271 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:25:37.271 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:25:37.281 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:37.282 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:25:37.283 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:25:37.289 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:37.293 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:25:37.299 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:37.310 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:25:37.312 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:25:37.319 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:25:37.323 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:25:37.333 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:25:37.337 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:37.340 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:25:37.349 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:37.351 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:25:37.367 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:25:37.368 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:25:37.369 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:37.379 INFO 4486 --- [host-check-host-status3,35e942a6cf1b120c,4c1fc941e2b0bb3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:25:37.379 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:25:37.392 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:25:37.397 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:25:37.405 INFO 4486 --- [host-check-host-status4,35e942a6cf1b120c,72d73e0d244a345f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:25:37.412 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:25:37.422 INFO 4486 --- [host-check-host-status2,35e942a6cf1b120c,1f528e8c4937a8ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:25:37.448 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:25:37.452 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:37.454 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.455 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:25:37.457 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.457 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.458 INFO 4486 --- [host-check-host-status1,35e942a6cf1b120c,9ccd615739d00c98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:25:37.458 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:25:37.469 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.469 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.469 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.469 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.469 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.470 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.498 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:25:37.536 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:25:37.582 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:37.599 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:37.601 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.602 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:25:37.603 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:25:37.604 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.605 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:25:37.606 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.606 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.607 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:25:37.616 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.616 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.616 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.616 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.616 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.617 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.625 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:25:37.807 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:25:37.853 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:37.855 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.857 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:25:37.858 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.859 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.859 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:25:37.870 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.870 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.870 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.870 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.870 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:37.870 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:37.889 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:25:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:25:37.993 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:25:37.994 INFO 4486 --- [prometheus-minute-schedule-1,400771dea3847dd6,8d65b360f2f9b158] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:38.001 INFO 4486 --- [prometheus-minute-schedule-1,400771dea3847dd6,8d65b360f2f9b158] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:38.108 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:25:38.160 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:38.162 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:38.163 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:25:38.164 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:38.165 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:38.166 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:25:38.175 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:38.175 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:38.175 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:38.175 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:38.175 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:25:38.176 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:38.236 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:25:38.425 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:25:38.239303, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:25:38.425 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:25:38.239303, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:25:38.462 INFO 4486 --- [ocp-schedules-29,e50b169cae682534,35d4790186f63e24] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8244} 2024-10-16 13:25:38.462 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:25:38.239303} } 2024-10-16 13:25:38.462 INFO 4486 --- [ocp-schedules-29,e50b169cae682534,35d4790186f63e24] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:25:38.523 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:25:38.531 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:25:38.534 INFO 4486 --- [ocp-schedules-30,ed0cfd27ec3e801e,be54c1e02003bd32] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:25:38.742 INFO 4486 --- [ocp-schedules-28,23ee35023c01fea2,b1d1c20d985a643d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:25:38.906 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,11da1ea77a50b69d,cfc96eb4eca3d713] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=11da1ea77a50b69d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:38.906 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,11da1ea77a50b69d,cfc96eb4eca3d713] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=11da1ea77a50b69d, duration=1 ms] 2024-10-16 13:25:38.967 INFO 4486 --- [ocp-async-3,308f4eeab90929db,753f1712761a6638] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:39.480 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:39.608 INFO 4486 --- [ocp-schedules-7,3446ba5018f7b8e1,5e317da345deb151] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:25:39.652 INFO 4486 --- [host-check-clockdiff3,a6406282d64b8928,de36b8ced0a26cff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056339 3 3 , error= 2024-10-16 13:25:39.653 INFO 4486 --- [host-check-clockdiff4,a6406282d64b8928,818acddc72f6d848] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056339 3 3 , error= 2024-10-16 13:25:39.665 WARN 4486 --- [scanner-13229736-1,3446ba5018f7b8e1,f8806dd9c03950ac] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:25:39.681 INFO 4486 --- [alarm-detect-16,3446ba5018f7b8e1,edb9e05b414c4a70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:25:39.685 INFO 4486 --- [alarm-detect-16,3446ba5018f7b8e1,edb9e05b414c4a70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:25:39.690 INFO 4486 --- [alarm-detect-16,3446ba5018f7b8e1,edb9e05b414c4a70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:25:39.691 INFO 4486 --- [alarm-detect-20,3446ba5018f7b8e1,0e01c86c8eba3063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:25:39.693 INFO 4486 --- [alarm-detect-5,3446ba5018f7b8e1,20a387e445100b02] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:25:39.696 INFO 4486 --- [alarm-detect-20,3446ba5018f7b8e1,0e01c86c8eba3063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:25:39.697 INFO 4486 --- [alarm-detect-5,3446ba5018f7b8e1,20a387e445100b02] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:25:39.700 INFO 4486 --- [alarm-detect-20,3446ba5018f7b8e1,0e01c86c8eba3063] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:25:39.701 INFO 4486 --- [alarm-detect-5,3446ba5018f7b8e1,20a387e445100b02] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:25:39.766 INFO 4486 --- [alarm-detect-12,3446ba5018f7b8e1,cf17f516ffaff856] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:25:39.772 INFO 4486 --- [alarm-detect-12,3446ba5018f7b8e1,cf17f516ffaff856] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:25:39.776 INFO 4486 --- [alarm-detect-12,3446ba5018f7b8e1,cf17f516ffaff856] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:25:40.007 INFO 4486 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:25:40.007 INFO 4486 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:25:40.019 INFO 4486 --- [ocp-schedules-28,7dc6d472c1ede8bf,bb47c5a89ecc8448] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:25:40.028 INFO 4486 --- [Check obproxy2,a472d9beb6cc66aa,5086f24a3614a14f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:25:40.028 INFO 4486 --- [Check obproxy1,8462aab622957768,b4b93acad5297378] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:25:40.028 INFO 4486 --- [Check obproxy2,a472d9beb6cc66aa,5086f24a3614a14f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:40.028 INFO 4486 --- [Check obproxy1,8462aab622957768,b4b93acad5297378] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:40.031 INFO 4486 --- [Collect obproxy2,efae489fd0356040,4dc9da8acac92fe1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:25:40.031 INFO 4486 --- [Collect obproxy2,efae489fd0356040,4dc9da8acac92fe1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:40.032 INFO 4486 --- [Collect obproxy1,00b5295c81276371,790994c61a81dbd9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:25:40.032 INFO 4486 --- [Collect obproxy1,00b5295c81276371,790994c61a81dbd9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:25:40.052 INFO 4486 --- [Check obproxy1,8462aab622957768,b4b93acad5297378] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:25:40.052 INFO 4486 --- [Check obproxy1,8462aab622957768,b4b93acad5297378] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:25:40.054 INFO 4486 --- [Check obproxy2,a472d9beb6cc66aa,5086f24a3614a14f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:25:40.054 INFO 4486 --- [Check obproxy2,a472d9beb6cc66aa,5086f24a3614a14f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:25:40.055 INFO 4486 --- [Collect obproxy2,efae489fd0356040,4dc9da8acac92fe1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:25:40.058 INFO 4486 --- [Collect obproxy1,00b5295c81276371,790994c61a81dbd9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:25:40.061 INFO 4486 --- [Collect obproxy2,efae489fd0356040,4dc9da8acac92fe1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:25:40.062 INFO 4486 --- [Collect obproxy1,00b5295c81276371,790994c61a81dbd9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:25:40.065 INFO 4486 --- [Collect obproxy1,00b5295c81276371,790994c61a81dbd9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:25:40.066 INFO 4486 --- [Collect obproxy2,efae489fd0356040,4dc9da8acac92fe1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:25:40.392 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b49cf1a1a6596eab,6f0a60d84867baf5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b49cf1a1a6596eab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:40.392 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b49cf1a1a6596eab,6f0a60d84867baf5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b49cf1a1a6596eab, duration=1 ms] 2024-10-16 13:25:40.967 INFO 4486 --- [ocp-async-6,b8728bd8c1bda2db,4402ef507cb699e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:40.977 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3078dd2f4388f52b,41113812eca1c984] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3078dd2f4388f52b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:40.977 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3078dd2f4388f52b,41113812eca1c984] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3078dd2f4388f52b, duration=1 ms] 2024-10-16 13:25:40.994 INFO 4486 --- [prometheus-minute-schedule-1,f33aa3cdf68c915d,cdaec7dee78bf9c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:40.995 INFO 4486 --- [prometheus-hour-schedule-1,63bd76291c70f907,05830f98ee7f6b46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:25:41.000 INFO 4486 --- [prometheus-minute-schedule-1,f33aa3cdf68c915d,cdaec7dee78bf9c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:42.056 INFO 4486 --- [ocp-schedules-15,364f296a1524f81a,be4cd24ba4de9546] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:25:42.481 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:42.967 INFO 4486 --- [ocp-async-5,faf14f8ebf980630,7f796727f8f1be8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:42.972 INFO 4486 --- [ocp-async-2,d2536869797649ed,fe8cd9c34fe42f86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:25:43.994 INFO 4486 --- [prometheus-minute-schedule-1,8e84ae9f7a0be79e,d4602fe9e3d4fc5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:44.001 INFO 4486 --- [prometheus-minute-schedule-1,8e84ae9f7a0be79e,d4602fe9e3d4fc5b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:44.088 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ebceb3c08f21b397,dba3220fd9ae2b07] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=ebceb3c08f21b397, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:25:44.088 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ebceb3c08f21b397,dba3220fd9ae2b07] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=ebceb3c08f21b397, duration=1 ms] 2024-10-16 13:25:44.968 INFO 4486 --- [ocp-async-8,dea636360d200174,b80bdbe45a9b97f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:45.399 INFO 4486 --- [ocp-schedules-5,abcbb785cf399db8,c110a3ea5b7f06de] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:25:45.405 INFO 4486 --- [ocp-schedules-5,abcbb785cf399db8,c110a3ea5b7f06de] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:25:45.405 INFO 4486 --- [ocp-schedules-5,abcbb785cf399db8,c110a3ea5b7f06de] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:25:45.413 INFO 4486 --- [ocp-schedules-30,ef355fad417bb2a3,8dfeaa846bc85b3d] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:25:45.413 INFO 4486 --- [ocp-schedules-30,ef355fad417bb2a3,8dfeaa846bc85b3d] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:25:45.482 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:46.744 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ced008257b302e3b,98e96f378c823185] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ced008257b302e3b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:46.744 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ced008257b302e3b,98e96f378c823185] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ced008257b302e3b, duration=1 ms] 2024-10-16 13:25:46.968 INFO 4486 --- [ocp-async-10,626da8fde1137f38,770df66ce2a7998c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:46.994 INFO 4486 --- [prometheus-minute-schedule-1,1ecd4d70a00c3936,0e516e041362e6b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:47.000 INFO 4486 --- [prometheus-minute-schedule-1,1ecd4d70a00c3936,0e516e041362e6b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:47.033 INFO 4486 --- [ocp-schedules-28,6a5e6329bbfe77ac,74d311483a197fb7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:25:47.070 INFO 4486 --- [ocp-schedules-11,71fb4d5bf32c19dc,f46696ef14293f34] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:25:47.182 INFO 4486 --- [ocp-schedules-22,bef9fb8f3f8cceeb,0d010c10c84fb55d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:25:48.483 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:48.758 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:25:48.912 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5ef6e913f31eb028,66bfcb25885d1c20] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5ef6e913f31eb028, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:48.912 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5ef6e913f31eb028,66bfcb25885d1c20] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5ef6e913f31eb028, duration=2 ms] 2024-10-16 13:25:48.968 INFO 4486 --- [ocp-async-5,fa90d98c2924c434,3345d8205daab440] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:49.421 INFO 4486 --- [ocp-schedules-28,1ede8c51dac21da6,e9fe544ca49b62d7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:25:49.421 INFO 4486 --- [ocp-schedules-28,1ede8c51dac21da6,e9fe544ca49b62d7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:25:49.421 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:25:49.421 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:25:49.421 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:25:49.421 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:25:49.421 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:25:49.427 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:49.427 INFO 4486 --- [ocp-schedules-28,1ede8c51dac21da6,e9fe544ca49b62d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:25:49.428 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:25:49.428 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:25:49.429 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:25:49.429 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:25:49.429 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:49.429 INFO 4486 --- [ocp-schedules-28,1ede8c51dac21da6,e9fe544ca49b62d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:25:49.429 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:25:49.430 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:25:49.430 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:25:49.430 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:25:49.456 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:25:49.456 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:25:49.457 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:25:49.457 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:25:49.465 INFO 4486 --- [ocp-schedules-16,95ac45397782ebdf,d6ef498246c64d0c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:25:49.465 INFO 4486 --- [ocp-schedules-28,1ede8c51dac21da6,e9fe544ca49b62d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:25:49.466 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:25:49.466 INFO 4486 --- [ocp-schedules-16,30052491b30ef974,27f209b7de3c0546] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:25:49.482 INFO 4486 --- [ocp-schedules-28,1ede8c51dac21da6,e9fe544ca49b62d7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:25:49.484 INFO 4486 --- [ocp-schedules-16,30052491b30ef974,27f209b7de3c0546] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:25:49.517 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.518 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:49.534 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.535 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:49.562 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.563 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:49.565 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:49.565 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.566 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:25:49.599 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:49.600 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.601 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:49.602 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:25:49.602 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.605 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:25:49.606 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:49.616 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:49.616 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:49.628 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:49.637 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:49.638 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.638 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:49.639 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:25:49.644 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:25:49.649 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:25:49.651 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:25:49.673 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:49.674 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.675 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:25:49.675 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:25:49.701 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:25:49.701 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:25:49.723 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:49.725 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.726 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:25:49.729 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.734 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:25:49.737 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:49.747 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:25:49.757 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:49.758 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:49.759 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:25:49.765 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:49.765 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:49.767 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:25:49.767 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:25:49.790 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:49.791 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:49.800 INFO 4486 --- [ocp-schedules-22,60881e0f7d1b04a6,92fb5877448fe453] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:25:49.812 WARN 4486 --- [scanner-13230018-1,60881e0f7d1b04a6,081b499475b40ede] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:25:49.818 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:49.818 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:49.847 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:49.847 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:25:49.892 INFO 4486 --- [alarm-detect-20,60881e0f7d1b04a6,244b1eebb07c3797] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056315 ~ 1729056339 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056330, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056330, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056330=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:25:49.943 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:49.943 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:25:49.994 INFO 4486 --- [prometheus-minute-schedule-1,916ea4b488ba765c,80cde542bab447e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:49.998 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:25:50.000 INFO 4486 --- [prometheus-minute-schedule-1,916ea4b488ba765c,80cde542bab447e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:50.024 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:50.026 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:50.026 INFO 4486 --- [ocp-schedules-15,cdfe74cb4a86d717,aa253491567363f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:25:50.027 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:25:50.031 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:25:50.036 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:25:50.040 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:25:50.042 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:25:50.066 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:25:50.205 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:25:50.235 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:25:50.237 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:25:50.238 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:25:50.267 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:25:50.267 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:25:50.315 INFO 4486 --- [ocp-schedules-7,64942289138c2524,3ae84f8819240bc9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:25:50.315 INFO 4486 --- [ocp-schedules-7,64942289138c2524,3ae84f8819240bc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:25:50.317 INFO 4486 --- [ocp-schedules-7,64942289138c2524,3ae84f8819240bc9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:25:50.317 INFO 4486 --- [ocp-schedules-7,64942289138c2524,3ae84f8819240bc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:25:50.318 INFO 4486 --- [ocp-schedules-7,64942289138c2524,3ae84f8819240bc9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:25:50.318 INFO 4486 --- [ocp-schedules-7,64942289138c2524,3ae84f8819240bc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703033600770641920 2024-10-16 13:25:50.397 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,627a43dff3e54447,8a418dbad5761bee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=627a43dff3e54447, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:50.397 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,627a43dff3e54447,8a418dbad5761bee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=627a43dff3e54447, duration=2 ms] 2024-10-16 13:25:50.969 INFO 4486 --- [ocp-async-9,1700beb0f7fd574f,3d85a211f78be4fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:50.983 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,47af0da22a99d7d7,fc5422b4e1849878] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=47af0da22a99d7d7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:50.983 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,47af0da22a99d7d7,fc5422b4e1849878] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=47af0da22a99d7d7, duration=1 ms] 2024-10-16 13:25:51.483 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:52.523 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9a6f762593a3df2c,95e852aa0eed5448] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=9a6f762593a3df2c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:25:52.524 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9a6f762593a3df2c,95e852aa0eed5448] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=9a6f762593a3df2c, duration=2 ms] 2024-10-16 13:25:52.969 INFO 4486 --- [ocp-async-1,f47f4a6a2d5f1a05,5c3281f4f816824c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:52.973 INFO 4486 --- [ocp-async-10,6d7092ef4fbac59e,c7d1ffe2e9444a08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:25:52.994 INFO 4486 --- [prometheus-minute-schedule-1,c0e3778e721d910b,09f5ca81bb720e8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:53.000 INFO 4486 --- [prometheus-minute-schedule-1,c0e3778e721d910b,09f5ca81bb720e8f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:54.484 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:54.969 INFO 4486 --- [ocp-async-4,fb1c5d66cdce2c09,e88c2435317da3d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:55.406 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:25:55.406 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:25:55.412 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:25:55.413 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:25:55.413 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:25:55.413 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:25:55.418 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:25:55.418 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:25:55.418 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:25:55.418 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:25:55.467 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:25:55.474 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:25:55.474 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:25:55.474 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:25:55.474 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:25:55.474 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:25:55.481 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:25:55.951 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:25:55.994 INFO 4486 --- [prometheus-minute-schedule-1,2a5120f02651ec8f,fbc6a9879ede80e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:56.001 INFO 4486 --- [prometheus-minute-schedule-1,2a5120f02651ec8f,fbc6a9879ede80e7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:56.435 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.016785566S 2024-10-16 13:25:56.435 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:25:56.500 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:25:56.500 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:25:56.523 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:25:56.594 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:25:56.594 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:25:56.594 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:25:56.594 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:25:56.594 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:25:56.594 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:25:56.603 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:25:56.746 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] o.s.boot.SpringApplication : Started application in 1.333 seconds (JVM running for 77259.797) 2024-10-16 13:25:56.746 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.334076208S 2024-10-16 13:25:56.749 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,66224217e7512d43,bd4c221e3aa86f5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=66224217e7512d43, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:56.749 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,66224217e7512d43,bd4c221e3aa86f5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=66224217e7512d43, duration=1 ms] 2024-10-16 13:25:56.969 INFO 4486 --- [ocp-async-9,2ab73cffd0147f35,2b4ce2f9708d47c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:57.144 INFO 4486 --- [ocp-schedules-9,f8b111dce4029cb4,c844fe5d8ad142e0] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:25:57.485 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:25:58.747 INFO 4486 --- [ocp-schedules-8,dc8be49f445d1b02,e6a65484f8475d77] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:25:58.918 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2ece9eaddb5287a8,7698dff4f7e17ac5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2ece9eaddb5287a8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:25:58.918 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2ece9eaddb5287a8,7698dff4f7e17ac5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2ece9eaddb5287a8, duration=2 ms] 2024-10-16 13:25:58.969 INFO 4486 --- [ocp-async-6,c3713e4133234e32,7e1fda15d8eb4cd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:25:58.994 INFO 4486 --- [prometheus-minute-schedule-1,0bd5c7e037f29a88,d8f01291cba98d9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056360, interval=60 2024-10-16 13:25:59.000 INFO 4486 --- [prometheus-minute-schedule-1,0bd5c7e037f29a88,d8f01291cba98d9f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:25:59.986 INFO 4486 --- [ocp-schedules-25,1d733507446f1e59,24d017a38660796a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:26:00.026 WARN 4486 --- [scanner-13230253-1,1d733507446f1e59,7997dd684a3c470b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:26:00.031 INFO 4486 --- [ocp-schedules-24,ce8bf9a9f43ea03c,bcc4f81c3385bf0e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:26:00.090 INFO 4486 --- [alarm-detect-15,1d733507446f1e59,cec913ad9d6c1797] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:00.097 INFO 4486 --- [alarm-detect-15,1d733507446f1e59,cec913ad9d6c1797] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:00.100 INFO 4486 --- [alarm-detect-15,1d733507446f1e59,cec913ad9d6c1797] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:00.102 INFO 4486 --- [alarm-detect-7,1d733507446f1e59,f7819369fc44ee25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:00.102 INFO 4486 --- [alarm-detect-14,1d733507446f1e59,b13c2e79f5604532] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:00.107 INFO 4486 --- [alarm-detect-7,1d733507446f1e59,f7819369fc44ee25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:00.107 INFO 4486 --- [alarm-detect-14,1d733507446f1e59,b13c2e79f5604532] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:00.111 INFO 4486 --- [alarm-detect-14,1d733507446f1e59,b13c2e79f5604532] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:00.111 INFO 4486 --- [alarm-detect-7,1d733507446f1e59,f7819369fc44ee25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:00.163 INFO 4486 --- [alarm-detect-11,1d733507446f1e59,305ba74f1bb7ecbd] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056330 ~ 1729056350 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056345, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056345, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056345=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:26:00.588 INFO 4486 --- [alarm-detect-11,1d733507446f1e59,43d5feb9f939bf1a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:00.588 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:00.592 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d3047b7ea9d22c3e,a758f292ccb137bb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d3047b7ea9d22c3e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:00.592 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d3047b7ea9d22c3e,a758f292ccb137bb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d3047b7ea9d22c3e, duration=3 ms] 2024-10-16 13:26:00.592 INFO 4486 --- [alarm-detect-11,1d733507446f1e59,43d5feb9f939bf1a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:00.598 INFO 4486 --- [alarm-detect-11,1d733507446f1e59,43d5feb9f939bf1a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:00.625 INFO 4486 --- [ocp-schedules-15,dfdd20028165cb40,f308f9089d45b4ae] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:26:00.625+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:26:00.628 INFO 4486 --- [ocp-schedules-15,dfdd20028165cb40,f308f9089d45b4ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:26:00.639 INFO 4486 --- [ocp-schedules-15,dfdd20028165cb40,f308f9089d45b4ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:26:00.651 INFO 4486 --- [ocp-schedules-15,dfdd20028165cb40,f308f9089d45b4ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:26:00.969 INFO 4486 --- [ocp-async-1,1b21018c990a3b80,cd356fcb49eefa4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:00.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e506e1e66b7cd82b,94d7e2cdafae475c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e506e1e66b7cd82b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:00.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e506e1e66b7cd82b,94d7e2cdafae475c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e506e1e66b7cd82b, duration=1 ms] 2024-10-16 13:26:01.994 INFO 4486 --- [prometheus-minute-schedule-1,966906f1efb03bef,0ddb70295757ea9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:02.000 INFO 4486 --- [prometheus-minute-schedule-1,966906f1efb03bef,0ddb70295757ea9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:02.970 INFO 4486 --- [ocp-async-7,09245e7b9c878996,12b936879301258f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:02.973 INFO 4486 --- [ocp-async-4,1536094baf8b8c8a,7639100a288b6bb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:26:03.589 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:04.970 INFO 4486 --- [ocp-async-9,ab2a0f8c0c34f31a,990b8a515e397128] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:04.994 INFO 4486 --- [prometheus-minute-schedule-1,a64f58c9ba893f5c,3f76c8ceee476742] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:05.002 INFO 4486 --- [prometheus-minute-schedule-1,a64f58c9ba893f5c,3f76c8ceee476742] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:06.590 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:06.757 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c249d22f568c395d,fd6d19d28771e39f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c249d22f568c395d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:06.757 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c249d22f568c395d,fd6d19d28771e39f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c249d22f568c395d, duration=3 ms] 2024-10-16 13:26:06.970 INFO 4486 --- [ocp-async-6,cb5015f92c9bfefd,d4397f00c6ab1cd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:07.066 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:26:07.067 INFO 4486 --- [ocp-schedules-23,ddfd4d2a8c1d29bd,a698be904cb1974e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:26:07.068 INFO 4486 --- [ocp-schedules-23,ddfd4d2a8c1d29bd,a698be904cb1974e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:07.081 INFO 4486 --- [ocp-schedules-10,745e59e8c7fbb2b4,4737317f02fc39d8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:26:07.083 INFO 4486 --- [ocp-schedules-10,745e59e8c7fbb2b4,4737317f02fc39d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:26:07.089 INFO 4486 --- [ocp-schedules-20,cdefee5659487bce,82558c9e23ab727c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:26:07.089 INFO 4486 --- [ocp-schedules-13,15663c4cbb3282dd,2bfebd2c9512583c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:26:07.089 INFO 4486 --- [ocp-schedules-29,040fcf2170660414,e10dd6a40aef2af7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:26:07.089 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:26:07.089 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:26:07.090 INFO 4486 --- [host-check-agent-status1,15663c4cbb3282dd,5dbc886b52c9cf02] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:26:07.090 INFO 4486 --- [host-check-clockdiff1,040fcf2170660414,cf64f9ec2483cb4b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:26:07.090 INFO 4486 --- [host-check-clockdiff2,040fcf2170660414,9af46738df780581] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:26:07.091 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:26:07.091 INFO 4486 --- [host-check-agent-status2,15663c4cbb3282dd,1200db0dbbeb5b30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:26:07.091 INFO 4486 --- [host-check-clockdiff3,040fcf2170660414,5e2a7a3531c0b54d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:26:07.092 INFO 4486 --- [host-check-agent-status3,15663c4cbb3282dd,403137a860b8f116] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:26:07.092 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:26:07.094 INFO 4486 --- [host-check-agent-status4,15663c4cbb3282dd,190c3136c0dbdf3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:26:07.094 INFO 4486 --- [host-check-clockdiff4,040fcf2170660414,f2e33d251d8b18a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:26:07.097 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:26:07.098 INFO 4486 --- [host-check-agent-status2,15663c4cbb3282dd,1200db0dbbeb5b30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:26:07.098 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:26:07.099 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:07.099 INFO 4486 --- [host-check-agent-status1,15663c4cbb3282dd,5dbc886b52c9cf02] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:07.099 INFO 4486 --- [host-check-agent-status3,15663c4cbb3282dd,403137a860b8f116] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:26:07.100 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:26:07.105 INFO 4486 --- [host-check-agent-status4,15663c4cbb3282dd,190c3136c0dbdf3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:26:07.130 INFO 4486 --- [host-check-clockdiff2,040fcf2170660414,9af46738df780581] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056367 3 3 , error= 2024-10-16 13:26:07.132 INFO 4486 --- [host-check-clockdiff4,040fcf2170660414,f2e33d251d8b18a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056367 3 3 , error= 2024-10-16 13:26:07.134 INFO 4486 --- [host-check-clockdiff1,040fcf2170660414,cf64f9ec2483cb4b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056367 0 0 , error= 2024-10-16 13:26:07.139 INFO 4486 --- [host-check-agent-status2,15663c4cbb3282dd,1200db0dbbeb5b30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:26:07.139 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:26:07.139 INFO 4486 --- [host-check-agent-status1,15663c4cbb3282dd,5dbc886b52c9cf02] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:07.139 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:07.139 INFO 4486 --- [host-check-agent-status3,15663c4cbb3282dd,403137a860b8f116] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:26:07.141 INFO 4486 --- [host-check-clockdiff3,040fcf2170660414,5e2a7a3531c0b54d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056367 3 3 , error= 2024-10-16 13:26:07.139 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:26:07.139 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:26:07.142 INFO 4486 --- [host-check-agent-status4,15663c4cbb3282dd,190c3136c0dbdf3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:26:07.152 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.152 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:07.154 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:26:07.170 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:26:07.170 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:26:07.171 INFO 4486 --- [host-check-agent-status1,15663c4cbb3282dd,5dbc886b52c9cf02] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:26:07.171 INFO 4486 --- [host-check-agent-status3,15663c4cbb3282dd,403137a860b8f116] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:26:07.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.171 INFO 4486 --- [host-check-agent-status4,15663c4cbb3282dd,190c3136c0dbdf3e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:26:07.172 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:26:07.172 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:07.174 INFO 4486 --- [host-check-agent-status2,15663c4cbb3282dd,1200db0dbbeb5b30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:26:07.174 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:26:07.174 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:26:07.176816857+08:00 2024-10-16 13:26:07.174 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:26:07.173493413+08:00 2024-10-16 13:26:07.175 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:26:07.177540+08:00 2024-10-16 13:26:07.175 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:26:07.174+08:00 2024-10-16 13:26:07.175 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:26:07.174+08:00 2024-10-16 13:26:07.175 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:26:07.175+08:00 2024-10-16 13:26:07.176 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:26:07.178576495+08:00 2024-10-16 13:26:07.176 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:26:07.176+08:00 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status3,15663c4cbb3282dd,403137a860b8f116] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status1,15663c4cbb3282dd,5dbc886b52c9cf02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status1,15663c4cbb3282dd,5dbc886b52c9cf02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status3,15663c4cbb3282dd,403137a860b8f116] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status1,15663c4cbb3282dd,5dbc886b52c9cf02] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status3,15663c4cbb3282dd,403137a860b8f116] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status4,15663c4cbb3282dd,190c3136c0dbdf3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status4,15663c4cbb3282dd,190c3136c0dbdf3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:26:07.177 INFO 4486 --- [host-check-agent-status4,15663c4cbb3282dd,190c3136c0dbdf3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:26:07.179 INFO 4486 --- [host-check-agent-status2,15663c4cbb3282dd,1200db0dbbeb5b30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:26:07.179 INFO 4486 --- [host-check-agent-status2,15663c4cbb3282dd,1200db0dbbeb5b30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:26:07.179 INFO 4486 --- [host-check-agent-status2,15663c4cbb3282dd,1200db0dbbeb5b30] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:26:07.185 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:26:07.175+08:00 and status AVAILABLE 2024-10-16 13:26:07.187 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:26:07.188 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:26:07.174+08:00 and status AVAILABLE 2024-10-16 13:26:07.188 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.189 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:26:07.176+08:00 and status AVAILABLE 2024-10-16 13:26:07.190 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:26:07.174+08:00 and status AVAILABLE 2024-10-16 13:26:07.190 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:26:07.192 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:26:07.192 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:26:07.192 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:26:07.196 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:26:07.198 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:26:07.198 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:07.200 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:07.201 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:26:07.203 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:26:07.203 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:26:07.204 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.205 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:26:07.205 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:07.214 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.215 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:07.223 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:07.224 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:07.224 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:07.227 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:07.227 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.228 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:07.230 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.231 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:26:07.246 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:07.247 INFO 4486 --- [ocp-schedules-22,b897a2e01489feba,43753941747a10a3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:26:07.256 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:26:07.260 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:26:07.262 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:26:07.269 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:26:07.273 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:26:07.276 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:26:07.277 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:26:07.288 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:26:07.300 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:07.302 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:07.304 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:07.305 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:07.332 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:26:07.335 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:26:07.343 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:26:07.344 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:26:07.345 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:26:07.354 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:26:07.360 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:07.361 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:07.362 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:26:07.370 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:07.371 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:26:07.386 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:07.389 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:26:07.397 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:26:07.400 INFO 4486 --- [host-check-host-status3,cdefee5659487bce,7d35098c3b113690] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:26:07.407 INFO 4486 --- [host-check-host-status4,cdefee5659487bce,5e637f4606762c7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:26:07.407 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:26:07.418 INFO 4486 --- [host-check-host-status2,cdefee5659487bce,59692ad3ffcab86c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:26:07.444 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:26:07.453 INFO 4486 --- [host-check-host-status1,cdefee5659487bce,29177c4d5f5ff274] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:26:07.994 INFO 4486 --- [prometheus-minute-schedule-1,6321a78b345e4929,5389e8d7a0e3cac3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:08.001 INFO 4486 --- [prometheus-minute-schedule-1,6321a78b345e4929,5389e8d7a0e3cac3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:08.925 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a49b14e69b39b7ec,6acef6b9e2685418] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a49b14e69b39b7ec, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:08.925 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a49b14e69b39b7ec,6acef6b9e2685418] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a49b14e69b39b7ec, duration=4 ms] 2024-10-16 13:26:08.970 INFO 4486 --- [ocp-async-7,a8ac7cb8aa8aa53a,ab6c5d91044fb4b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:09.591 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:10.007 INFO 4486 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:26:10.008 INFO 4486 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:26:10.028 INFO 4486 --- [Check obproxy1,9617f99c9df5d7d2,4da1f4e97c619c62] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:26:10.028 INFO 4486 --- [Check obproxy2,d75ce761bea14253,893591de362cb403] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:26:10.033 INFO 4486 --- [Collect obproxy2,e5e1bbb0e21d805b,909ab8e69374097d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:26:10.034 INFO 4486 --- [Collect obproxy1,08887579b0862e8a,6e50df762cdd91dd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:26:10.036 INFO 4486 --- [Check obproxy1,9617f99c9df5d7d2,4da1f4e97c619c62] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:10.038 INFO 4486 --- [Check obproxy2,d75ce761bea14253,893591de362cb403] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:10.039 INFO 4486 --- [Collect obproxy2,e5e1bbb0e21d805b,909ab8e69374097d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:10.041 INFO 4486 --- [Collect obproxy1,08887579b0862e8a,6e50df762cdd91dd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:10.044 INFO 4486 --- [ocp-schedules-29,4b5b6f98b0c09ba4,2a260095488e5162] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:26:10.063 INFO 4486 --- [Check obproxy2,d75ce761bea14253,893591de362cb403] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:26:10.063 INFO 4486 --- [Check obproxy2,d75ce761bea14253,893591de362cb403] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:26:10.064 INFO 4486 --- [Collect obproxy2,e5e1bbb0e21d805b,909ab8e69374097d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:26:10.068 INFO 4486 --- [Collect obproxy2,e5e1bbb0e21d805b,909ab8e69374097d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:26:10.068 INFO 4486 --- [Collect obproxy1,08887579b0862e8a,6e50df762cdd91dd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:26:10.069 INFO 4486 --- [Check obproxy1,9617f99c9df5d7d2,4da1f4e97c619c62] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:26:10.069 INFO 4486 --- [Check obproxy1,9617f99c9df5d7d2,4da1f4e97c619c62] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:26:10.071 INFO 4486 --- [Collect obproxy1,08887579b0862e8a,6e50df762cdd91dd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:26:10.072 INFO 4486 --- [Collect obproxy2,e5e1bbb0e21d805b,909ab8e69374097d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:26:10.076 INFO 4486 --- [Collect obproxy1,08887579b0862e8a,6e50df762cdd91dd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:26:10.596 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ac7cfc738d34179b,50e58b5abc6ad6a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ac7cfc738d34179b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:10.596 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ac7cfc738d34179b,50e58b5abc6ad6a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ac7cfc738d34179b, duration=0 ms] 2024-10-16 13:26:10.615 INFO 4486 --- [ocp-schedules-6,f7b3e9ecaea46428,02370b2f6ef428ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:26:10.666 WARN 4486 --- [scanner-13230608-1,f7b3e9ecaea46428,b39816b8f990286e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:26:10.705 INFO 4486 --- [alarm-detect-10,f7b3e9ecaea46428,4b07dc059b10d23f] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056345 ~ 1729056360 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056360, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056360, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056360=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:26:10.971 INFO 4486 --- [ocp-async-5,8b22edca8b5181e2,e3e8cfc20494a9bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:10.980 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:26:10.992 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c39ef6201cd09a4e,bf409173b8a7fc38] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c39ef6201cd09a4e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:10.992 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c39ef6201cd09a4e,bf409173b8a7fc38] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c39ef6201cd09a4e, duration=1 ms] 2024-10-16 13:26:10.994 INFO 4486 --- [prometheus-minute-schedule-1,dd76bb8d6a563295,7da0b9c9c9ba977e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:10.999 INFO 4486 --- [prometheus-minute-schedule-1,dd76bb8d6a563295,7da0b9c9c9ba977e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:26:12.592 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:12.825 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=79, affectRows=79 2024-10-16 13:26:12.826 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=79, validCount=79 2024-10-16 13:26:12.826 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=96, affectRows=96 2024-10-16 13:26:12.827 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=93, affectRows=93 2024-10-16 13:26:12.827 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=95, validCount=95 2024-10-16 13:26:12.827 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=56, affectRows=56 2024-10-16 13:26:12.827 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=93, validCount=93 2024-10-16 13:26:12.827 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=56, validCount=56 2024-10-16 13:26:12.828 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=125, affectRows=125 2024-10-16 13:26:12.828 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=124, validCount=124 2024-10-16 13:26:12.844 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=1000, affectRows=1000 2024-10-16 13:26:12.847 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=999, validCount=999 2024-10-16 13:26:12.859 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=1000, affectRows=1000 2024-10-16 13:26:12.862 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056300, nodeCount=1000, affectRows=1000 2024-10-16 13:26:12.863 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=1000, validCount=1000 2024-10-16 13:26:12.866 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056300, nodeCount=1000, validCount=1000 2024-10-16 13:26:12.971 INFO 4486 --- [ocp-async-8,eeeb5722d3657da3,f31c7a4b23a064d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:12.973 INFO 4486 --- [ocp-async-6,fbf7c4e9106c5b7e,c0b7ab931c686dc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:26:13.429 INFO 4486 --- [ocp-schedules-30,d8f6cad025cc0147,804c6a996b774781] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:26:13.432 INFO 4486 --- [ocp-schedules-30,d8f6cad025cc0147,804c6a996b774781] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:13.433 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:26:13.433 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:26:13.433 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:26:13.994 INFO 4486 --- [prometheus-minute-schedule-1,bd3158a9e76de410,dcd390c87f02a89f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:14.000 INFO 4486 --- [prometheus-minute-schedule-1,bd3158a9e76de410,dcd390c87f02a89f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:14.259 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056300, nodeCount=477, affectRows=477 2024-10-16 13:26:14.910 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056300, nodeCount=477, validCount=477 2024-10-16 13:26:14.971 INFO 4486 --- [ocp-async-7,a2d2d51478f5fc20,d1314b0821eee0d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:15.593 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:16.765 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,229563f95cd53427,88ed1e9b8325209e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=229563f95cd53427, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:16.766 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,229563f95cd53427,88ed1e9b8325209e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=229563f95cd53427, duration=3 ms] 2024-10-16 13:26:16.972 INFO 4486 --- [ocp-async-3,e924cbfa7d9bc071,daf27464a8a377b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:16.994 INFO 4486 --- [prometheus-minute-schedule-1,fb9465d1fc8e6fcd,f41a8f99a47030c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:16.999 INFO 4486 --- [prometheus-minute-schedule-1,fb9465d1fc8e6fcd,f41a8f99a47030c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:17.037 INFO 4486 --- [ocp-schedules-1,51ec1214302074fe,95c5dea7f96a6172] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:26:17.069 INFO 4486 --- [ocp-schedules-23,ddfd4d2a8c1d29bd,a698be904cb1974e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:26:17.177 INFO 4486 --- [ocp-schedules-10,745e59e8c7fbb2b4,4737317f02fc39d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:26:18.594 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:18.752 INFO 4486 --- [ocp-schedules-27,bb415b8be93907ca,342956d8474f71dd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:26:18.929 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b81135d341bd2b44,eef8c55faac90a36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b81135d341bd2b44, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:18.929 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b81135d341bd2b44,eef8c55faac90a36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b81135d341bd2b44, duration=1 ms] 2024-10-16 13:26:18.972 INFO 4486 --- [ocp-async-9,9ed4a1976cb46cba,bfbd80b8de2e1395] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:19.994 INFO 4486 --- [prometheus-minute-schedule-1,9426d9d4c40cb1b5,2e3d102ed7ea5ab5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:19.999 INFO 4486 --- [prometheus-minute-schedule-1,9426d9d4c40cb1b5,2e3d102ed7ea5ab5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:20.050 INFO 4486 --- [ocp-schedules-13,969809e6d82e3a99,5065b5c9483ea0f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:26:20.267 INFO 4486 --- [ocp-schedules-16,30052491b30ef974,27f209b7de3c0546] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:26:20.268 INFO 4486 --- [ocp-schedules-16,e67947447464ea4b,9d7ad297c5154c4c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:26:20.270 INFO 4486 --- [ocp-schedules-16,e67947447464ea4b,9d7ad297c5154c4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:20.304 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.305 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:20.310 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.311 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:20.333 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.334 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:20.340 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.345 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.345 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:26:20.379 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.380 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.380 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:20.382 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:26:20.386 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.389 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:26:20.391 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:20.403 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.404 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:20.419 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.427 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.429 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.430 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:20.431 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:26:20.438 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:26:20.443 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:20.445 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:20.449 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.450 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.451 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:26:20.473 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:26:20.481 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:26:20.481 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:26:20.497 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.499 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.500 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:20.503 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.506 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:26:20.509 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:20.536 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.536 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:20.547 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:26:20.557 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.562 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.562 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:26:20.563 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.563 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:20.572 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:26:20.572 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:26:20.592 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.592 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:20.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4f27c47e7e0d2168,45c7be7c727a8c6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4f27c47e7e0d2168, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:20.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4f27c47e7e0d2168,45c7be7c727a8c6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4f27c47e7e0d2168, duration=2 ms] 2024-10-16 13:26:20.624 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.624 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:20.653 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.653 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:20.681 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:20.707 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.709 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.710 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:20.714 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:26:20.719 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:26:20.723 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:26:20.725 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:26:20.749 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:26:20.768 INFO 4486 --- [ocp-schedules-24,dad53ad5214819c8,11b71f7270960a9d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:26:20.809 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:26:20.817 WARN 4486 --- [scanner-13231676-1,dad53ad5214819c8,bfff961cb7349c4f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:26:20.882 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:26:20.895 INFO 4486 --- [alarm-detect-10,dad53ad5214819c8,7579644d5b2e258f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:20.896 INFO 4486 --- [alarm-detect-8,dad53ad5214819c8,7343806006ffc207] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:20.897 INFO 4486 --- [alarm-detect-20,dad53ad5214819c8,00c693cd562be236] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:20.903 INFO 4486 --- [alarm-detect-10,dad53ad5214819c8,7579644d5b2e258f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:20.904 INFO 4486 --- [alarm-detect-20,dad53ad5214819c8,00c693cd562be236] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:20.906 INFO 4486 --- [alarm-detect-8,dad53ad5214819c8,7343806006ffc207] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:20.909 INFO 4486 --- [alarm-detect-20,dad53ad5214819c8,00c693cd562be236] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:20.911 INFO 4486 --- [alarm-detect-10,dad53ad5214819c8,7579644d5b2e258f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:20.913 INFO 4486 --- [alarm-detect-8,dad53ad5214819c8,7343806006ffc207] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:20.954 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:20.962 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:20.965 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:26:20.972 INFO 4486 --- [ocp-async-10,b0b0ae6909ce9b60,1528b53f998fe7f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:21.000 INFO 4486 --- [prometheus-cache-monitor1,97a5833e839209d4,7c6640247f79a583] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3414,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8244} 2024-10-16 13:26:21.000 INFO 4486 --- [prometheus-cache-monitor1,97a5833e839209d4,7c6640247f79a583] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:26:21.001 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,46c2fee7cce75acd,b8689f9afdc4e147] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=46c2fee7cce75acd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:21.002 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,46c2fee7cce75acd,b8689f9afdc4e147] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=46c2fee7cce75acd, duration=2 ms] 2024-10-16 13:26:21.002 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:26:21.002 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:26:21.023 INFO 4486 --- [alarm-detect-16,dad53ad5214819c8,7539ac7ac830417c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:21.028 INFO 4486 --- [alarm-detect-16,dad53ad5214819c8,7539ac7ac830417c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:21.032 INFO 4486 --- [alarm-detect-16,dad53ad5214819c8,7539ac7ac830417c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:21.595 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:22.481 INFO 4486 --- [ocp-schedules-22,4aa97a1a0f430b97,ed8bfaf1359ec1ca] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:26:22.678 INFO 4486 --- [ocp-schedules-14,aa9aaab9eba74843,0ef0d135cf2702b3] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:26:22.680 INFO 4486 --- [ocp-schedules-14,aa9aaab9eba74843,0ef0d135cf2702b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:22.713 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:26:22.714 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.718 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:26:22.725 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.726 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:26:22.727 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:26:22.727 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.731 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:26:22.734 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.734 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:26:22.742 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.743 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:26:22.746 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.746 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:26:22.754 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.754 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:26:22.763 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:26:22.764 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.765 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.766 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:26:22.766 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.767 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:22.769 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:26:22.778 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.779 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:22.782 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.782 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.782 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:22.783 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:26:22.795 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.796 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:26:22.802 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.803 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:22.810 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.811 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:26:22.814 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.814 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:26:22.816 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.817 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:26:22.828 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.829 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:26:22.830 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.830 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:26:22.833 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.834 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:26:22.845 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.846 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.847 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.847 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.847 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.848 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.848 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:26:22.849 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.849 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.850 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.851 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.852 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.852 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.853 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.854 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.854 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.855 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.856 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.856 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.857 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.858 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.858 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.859 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.859 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.859 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.860 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.860 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.860 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.861 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:26:22.861 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.862 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:26:22.863 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.864 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:22.869 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.870 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.871 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:26:22.872 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.873 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.874 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.875 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:22.875 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.876 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.880 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.882 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.885 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.885 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.887 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.887 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:22.887 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:22.888 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.888 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:26:22.891 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.892 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.896 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.897 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.901 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.902 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.903 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:22.904 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:26:22.906 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:26:22.909 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:26:22.909 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:26:22.916 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.918 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.921 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.922 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.925 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.926 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.928 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.929 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.932 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:22.932 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.933 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.933 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.934 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:26:22.936 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.937 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.939 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.940 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:26:22.943 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.944 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:26:22.945 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:22.945 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:26:22.945 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:22.945 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:26:22.945 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:22.945 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:26:22.947 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:22.948 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:26:22.948 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:26:22.948 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:26:22.949 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:26:22.951 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:26:22.972 INFO 4486 --- [ocp-async-4,6be69ab7bacac6ad,186a3c0b0afb77e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:22.973 INFO 4486 --- [ocp-async-5,d93fb2fb4b81f434,dcee2c7d6c2772e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:26:22.994 INFO 4486 --- [prometheus-minute-schedule-1,9f830b9d13e50039,723abcfeb1d81c2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:22.999 INFO 4486 --- [prometheus-minute-schedule-1,9f830b9d13e50039,723abcfeb1d81c2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:23.009 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:23.010 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:23.011 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:26:23.021 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:23.021 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:26:23.021 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:23.021 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:26:23.021 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:26:23.021 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:26:23.024 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:26:23.025 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:26:23.433 INFO 4486 --- [ocp-schedules-30,d8f6cad025cc0147,804c6a996b774781] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:26:23.921 INFO 4486 --- [ocp-schedules-25,085dada47220b218,d317b15701bc7979] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:26:23.921 INFO 4486 --- [ocp-schedules-25,085dada47220b218,d317b15701bc7979] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:26:23.931 INFO 4486 --- [ocp-schedules-7,53c1313d00a622a1,90454b2763d8b4d7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:26:23.963 INFO 4486 --- [Collect all tenants major compactions1,085dada47220b218,2c980f252c116a37] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:26:23.963 INFO 4486 --- [Collect all tenants major compactions2,085dada47220b218,6c33af3861170699] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:26:23.963 INFO 4486 --- [Collect all tenants major compactions3,085dada47220b218,431486b34b80dbe1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:26:23.963 INFO 4486 --- [Collect all tenants major compactions4,085dada47220b218,de0cc9c5c4f7d5bd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:26:23.964 INFO 4486 --- [Collect all tenants major compactions5,085dada47220b218,c1d17a7458fea802] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:26:23.985 INFO 4486 --- [Collect all tenants major compactions1,085dada47220b218,2c980f252c116a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:23.985 INFO 4486 --- [Collect all tenants major compactions3,085dada47220b218,431486b34b80dbe1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:23.986 INFO 4486 --- [Collect all tenants major compactions1,085dada47220b218,2c980f252c116a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:26:23.986 INFO 4486 --- [Collect all tenants major compactions2,085dada47220b218,6c33af3861170699] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:23.987 INFO 4486 --- [Collect all tenants major compactions3,085dada47220b218,431486b34b80dbe1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:26:23.989 INFO 4486 --- [Collect all tenants major compactions2,085dada47220b218,6c33af3861170699] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:26:23.990 INFO 4486 --- [Collect all tenants major compactions1,085dada47220b218,2c980f252c116a37] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:26:23.992 INFO 4486 --- [Collect all tenants major compactions2,085dada47220b218,6c33af3861170699] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:26:23.992 INFO 4486 --- [Collect all tenants major compactions3,085dada47220b218,431486b34b80dbe1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:26:23.995 INFO 4486 --- [Collect all tenants major compactions1,085dada47220b218,2c980f252c116a37] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:26:23.999 INFO 4486 --- [Collect all tenants major compactions2,085dada47220b218,6c33af3861170699] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:26:23.999 INFO 4486 --- [Collect all tenants major compactions4,085dada47220b218,de0cc9c5c4f7d5bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:24.001 INFO 4486 --- [Collect all tenants major compactions3,085dada47220b218,431486b34b80dbe1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:26:24.001 INFO 4486 --- [Collect all tenants major compactions5,085dada47220b218,c1d17a7458fea802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:24.001 INFO 4486 --- [Collect all tenants major compactions4,085dada47220b218,de0cc9c5c4f7d5bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:26:24.004 INFO 4486 --- [Collect all tenants major compactions5,085dada47220b218,c1d17a7458fea802] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:26:24.006 INFO 4486 --- [Collect all tenants major compactions4,085dada47220b218,de0cc9c5c4f7d5bd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:26:24.006 INFO 4486 --- [Collect all tenants major compactions4,085dada47220b218,de0cc9c5c4f7d5bd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:26:24.009 INFO 4486 --- [Collect all tenants major compactions5,085dada47220b218,c1d17a7458fea802] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:26:24.009 INFO 4486 --- [Collect all tenants major compactions5,085dada47220b218,c1d17a7458fea802] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:26:24.011 INFO 4486 --- [ocp-schedules-25,085dada47220b218,d317b15701bc7979] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:26:24.400 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-10-16 13:26:24.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=observer, size=5 2024-10-16 13:26:24.404 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=obproxy, size=3 2024-10-16 13:26:24.405 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=observer, size=5 2024-10-16 13:26:24.406 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=obproxy, size=3 2024-10-16 13:26:24.408 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=observer, size=5 2024-10-16 13:26:24.409 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=obproxy, size=3 2024-10-16 13:26:24.410 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=observer, size=5 2024-10-16 13:26:24.411 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=obproxy, size=3 2024-10-16 13:26:24.413 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=observer, size=5 2024-10-16 13:26:24.415 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=obproxy, size=3 2024-10-16 13:26:24.416 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=observer, size=5 2024-10-16 13:26:24.417 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=obproxy, size=3 2024-10-16 13:26:24.419 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=observer, size=5 2024-10-16 13:26:24.420 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=obproxy, size=3 2024-10-16 13:26:24.420 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-10-16 13:26:24.420 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=40024 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703033600594481152 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703033600598675456 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703033600611258369 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703033600611258368 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703033600607064065 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703033600607064064 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703033600628035585 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703033600628035584 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703033600632229888 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703033600619646977 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703033600619646976 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703033600644812804 2024-10-16 13:26:24.557 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703033600644812801 2024-10-16 13:26:24.558 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703033600644812800 2024-10-16 13:26:24.558 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703033600644812803 2024-10-16 13:26:24.558 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703033600644812802 2024-10-16 13:26:24.558 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703033600649007104 2024-10-16 13:26:24.558 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703033600636424193 2024-10-16 13:26:24.558 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703033600636424192 2024-10-16 13:26:24.558 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703033600640618496 2024-10-16 13:26:24.596 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:24.973 INFO 4486 --- [ocp-async-8,7bcce2a4ed58e23b,d1c26d250c153c0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703033600728698880 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703033600732893185 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703033600732893184 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703033600724504576 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703033600749670400 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703033600741281792 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703033600762253312 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703033600753864704 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703033600758059008 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703033600770641920 2024-10-16 13:26:25.402 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703033600665784321 2024-10-16 13:26:25.403 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703033600665784320 2024-10-16 13:26:25.403 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703033600665784322 2024-10-16 13:26:25.403 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703033600653201409 2024-10-16 13:26:25.403 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703033600653201408 2024-10-16 13:26:25.403 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703033600653201411 2024-10-16 13:26:25.403 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703033600653201410 2024-10-16 13:26:25.403 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703033600657395712 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703033600678367233 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703033600678367232 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703033600678367235 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703033600678367234 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703033600682561537 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703033600682561536 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703033600669978625 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703033600669978624 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703033600669978626 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703033600674172928 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703033600686755841 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703033600686755840 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703033600690950145 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703033600690950144 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703033600711921664 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703033600716115968 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703033600703533056 2024-10-16 13:26:25.456 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703033600707727360 2024-10-16 13:26:25.994 INFO 4486 --- [prometheus-minute-schedule-1,eb01412503426657,f7a627a8832fd005] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:26.000 INFO 4486 --- [prometheus-minute-schedule-1,eb01412503426657,f7a627a8832fd005] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:26.601 INFO 4486 --- [ocp-schedules-4,5cc7a80a21aced78,0551bda83ae260f2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:26:26.601+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:26:26.605 INFO 4486 --- [ocp-schedules-4,5cc7a80a21aced78,0551bda83ae260f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:26:26.606 INFO 4486 --- [ocp-schedules-4,5cc7a80a21aced78,0551bda83ae260f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:26:26.607 INFO 4486 --- [ocp-schedules-4,5cc7a80a21aced78,0551bda83ae260f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:26:26.748 INFO 4486 --- [ocp-schedules-27,456e0fec9f03ffc1,a7c41d9c36cdfd30] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=39968 2024-10-16 13:26:26.772 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ca9d4cea95331e27,dd696440183a4692] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ca9d4cea95331e27, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:26.772 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ca9d4cea95331e27,dd696440183a4692] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ca9d4cea95331e27, duration=2 ms] 2024-10-16 13:26:26.973 INFO 4486 --- [ocp-async-2,2c70216c22bbfc63,548996499cef7066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:27.597 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:27.867 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ab85f479e6e070d8,c2fddda599762319] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=ab85f479e6e070d8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:26:27.868 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ab85f479e6e070d8,c2fddda599762319] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=ab85f479e6e070d8, duration=2 ms] 2024-10-16 13:26:28.933 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,51757fcf978e12d2,3faa00dbc450a6ec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=51757fcf978e12d2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:28.934 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,51757fcf978e12d2,3faa00dbc450a6ec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=51757fcf978e12d2, duration=2 ms] 2024-10-16 13:26:28.973 INFO 4486 --- [ocp-async-7,614f21bd4898556b,174334a1eaeb6584] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:28.994 INFO 4486 --- [prometheus-minute-schedule-1,dca0b19bfe33e8fb,3aa0b34ba6321a1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:28.999 INFO 4486 --- [prometheus-minute-schedule-1,dca0b19bfe33e8fb,3aa0b34ba6321a1e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:30.055 INFO 4486 --- [ocp-schedules-11,af064e41fda462fe,20aac2a2be7436d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:26:30.103 INFO 4486 --- [prometheus-second-schedule-1,58691f9f11450385,19eae4ed72d9a150] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:26:30.598 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:30.607 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9e4b4a8e09bc9abd,66cb54f193a3047d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9e4b4a8e09bc9abd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:30.607 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9e4b4a8e09bc9abd,66cb54f193a3047d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9e4b4a8e09bc9abd, duration=1 ms] 2024-10-16 13:26:30.973 INFO 4486 --- [ocp-async-4,5f03141980e97ed6,08211eabbe5383c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:31.007 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,644d223fc015b998,a72b55722800ab69] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=644d223fc015b998, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:31.007 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,644d223fc015b998,a72b55722800ab69] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=644d223fc015b998, duration=1 ms] 2024-10-16 13:26:31.091 INFO 4486 --- [ocp-schedules-13,d9d0bb492eff8c87,bd0cfaf26fc9e274] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:26:31.118 WARN 4486 --- [scanner-13232015-1,d9d0bb492eff8c87,405d878a971a0a32] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:26:31.178 INFO 4486 --- [alarm-detect-10,d9d0bb492eff8c87,d81b0a6b5ca9dda9] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056360 ~ 1729056381 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056375, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056375, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056375=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:26:31.994 INFO 4486 --- [prometheus-minute-schedule-1,779c981c88f71e0a,9565ffbdfd6948f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:31.999 INFO 4486 --- [prometheus-minute-schedule-1,779c981c88f71e0a,9565ffbdfd6948f9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:32.346 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e42f8193710d6ed5,1718c35a76da8477] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=e42f8193710d6ed5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:26:32.346 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e42f8193710d6ed5,1718c35a76da8477] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=e42f8193710d6ed5, duration=1 ms] 2024-10-16 13:26:32.974 INFO 4486 --- [ocp-async-9,15c526868c4657f4,8fece64d1514fe12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:26:32.974 INFO 4486 --- [ocp-async-6,d0989ee059a2040d,99a2cbdf988cc051] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:33.599 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:33.911 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1448213dd57d532c,9c9320b3831b47b8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=1448213dd57d532c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:26:33.911 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1448213dd57d532c,9c9320b3831b47b8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=1448213dd57d532c, duration=2 ms] 2024-10-16 13:26:34.974 INFO 4486 --- [ocp-async-7,27e95926fabdb6cc,690ee4e7e3469f1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:34.994 INFO 4486 --- [prometheus-minute-schedule-1,406406a634387ae8,822c2942b6450c3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:34.999 INFO 4486 --- [prometheus-minute-schedule-1,406406a634387ae8,822c2942b6450c3a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:36.600 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:36.778 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6ea11896292221bb,d4176ced36e4fddb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6ea11896292221bb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:36.779 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6ea11896292221bb,d4176ced36e4fddb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6ea11896292221bb, duration=1 ms] 2024-10-16 13:26:36.974 INFO 4486 --- [ocp-async-4,cd120dccd5a92605,e0bd50fc23638fdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:37.063 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:26:37.064 INFO 4486 --- [ocp-schedules-12,1e9b627285d778da,4ce783a635f358d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:26:37.066 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:26:37.067 INFO 4486 --- [ocp-schedules-10,77da91f06998ce6e,d4c12cfe20457c6a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:26:37.067 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:26:37.069 INFO 4486 --- [ocp-schedules-10,77da91f06998ce6e,d4c12cfe20457c6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:37.072 INFO 4486 --- [ocp-schedules-12,1e9b627285d778da,4ce783a635f358d4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:26:37.081 INFO 4486 --- [ocp-schedules-8,32d4d7720ab342d4,b227d3de4a4a33d6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:26:37.083 INFO 4486 --- [ocp-schedules-8,32d4d7720ab342d4,b227d3de4a4a33d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:37.083 INFO 4486 --- [ocp-schedules-6,1ef226ea17f7e19b,7af5d25f2b4317d7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 13:26:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:26:37.085 INFO 4486 --- [ocp-schedules-6,1ef226ea17f7e19b,7af5d25f2b4317d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:37.090 INFO 4486 --- [ocp-schedules-25,79b25de477156ad0,a86b313664e76b3b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:26:37.091 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:26:37.092 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:26:37.092 INFO 4486 --- [ocp-schedules-5,776669e1dad7a862,ac5877742c9e7d04] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:26:37.092 INFO 4486 --- [ocp-schedules-20,fdc3730f499ab8dd,cecf4e388a26593b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:26:37.092 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:26:37.092 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:26:37.094 INFO 4486 --- [host-check-clockdiff2,776669e1dad7a862,53969aab6ba7d6c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:26:37.094 INFO 4486 --- [host-check-clockdiff3,776669e1dad7a862,6efd664a02743d54] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:26:37.094 INFO 4486 --- [host-check-clockdiff1,776669e1dad7a862,130c26c11afba993] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:26:37.095 INFO 4486 --- [host-check-agent-status1,fdc3730f499ab8dd,9cf28b36398d8f1b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:26:37.096 INFO 4486 --- [host-check-agent-status3,fdc3730f499ab8dd,4a306fbc9b5e84ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:26:37.096 INFO 4486 --- [host-check-agent-status4,fdc3730f499ab8dd,f65a4d4f37376473] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:26:37.096 INFO 4486 --- [host-check-agent-status2,fdc3730f499ab8dd,debb4efdd9cbd365] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:26:37.097 INFO 4486 --- [host-check-clockdiff4,776669e1dad7a862,4f05ebdd051a8232] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:26:37.109 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:37.110 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:26:37.110 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:26:37.110 INFO 4486 --- [host-check-agent-status1,fdc3730f499ab8dd,9cf28b36398d8f1b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:37.110 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:26:37.112 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:26:37.112 INFO 4486 --- [host-check-agent-status1,fdc3730f499ab8dd,9cf28b36398d8f1b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:37.113 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:26:37.112 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:26:37.112 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:37.113 INFO 4486 --- [host-check-agent-status4,fdc3730f499ab8dd,f65a4d4f37376473] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:26:37.124 INFO 4486 --- [host-check-agent-status2,fdc3730f499ab8dd,debb4efdd9cbd365] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:26:37.134 INFO 4486 --- [host-check-agent-status3,fdc3730f499ab8dd,4a306fbc9b5e84ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:26:37.136 INFO 4486 --- [host-check-clockdiff2,776669e1dad7a862,53969aab6ba7d6c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056397 3 3 , error= 2024-10-16 13:26:37.145 INFO 4486 --- [host-check-clockdiff1,776669e1dad7a862,130c26c11afba993] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056397 0 0 , error= 2024-10-16 13:26:37.147 INFO 4486 --- [host-check-clockdiff4,776669e1dad7a862,4f05ebdd051a8232] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056397 3 3 , error= 2024-10-16 13:26:37.150 INFO 4486 --- [host-check-clockdiff3,776669e1dad7a862,6efd664a02743d54] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056397 3 3 , error= 2024-10-16 13:26:37.151 INFO 4486 --- [host-check-agent-status4,fdc3730f499ab8dd,f65a4d4f37376473] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:26:37.151 INFO 4486 --- [host-check-agent-status3,fdc3730f499ab8dd,4a306fbc9b5e84ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:26:37.152 INFO 4486 --- [host-check-agent-status2,fdc3730f499ab8dd,debb4efdd9cbd365] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:26:37.152 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:37.153 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.154 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:26:37.154 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:26:37.168 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9a8b425b0afeb198,ff3ac7981c9fedb4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=9a8b425b0afeb198, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 13:26:37.168 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9a8b425b0afeb198,ff3ac7981c9fedb4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=9a8b425b0afeb198, duration=2 ms] 2024-10-16 13:26:37.171 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.172 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:26:37.177 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:26:37.178 INFO 4486 --- [host-check-agent-status1,fdc3730f499ab8dd,9cf28b36398d8f1b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:26:37.178 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.179 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.179 INFO 4486 --- [host-check-agent-status4,fdc3730f499ab8dd,f65a4d4f37376473] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:26:37.179 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:26:37.179 INFO 4486 --- [host-check-agent-status3,fdc3730f499ab8dd,4a306fbc9b5e84ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:26:37.179 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:26:37.180 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:37.180 INFO 4486 --- [host-check-agent-status2,fdc3730f499ab8dd,debb4efdd9cbd365] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:26:37.181 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:26:37.181 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.182 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:26:37.184157969+08:00 2024-10-16 13:26:37.182 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:26:37.184786292+08:00 2024-10-16 13:26:37.182 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:26:37.182+08:00 2024-10-16 13:26:37.183 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:26:37.183 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:26:37.182+08:00 2024-10-16 13:26:37.183 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:26:37.185845839+08:00 2024-10-16 13:26:37.184 INFO 4486 --- [host-check-agent-status3,fdc3730f499ab8dd,4a306fbc9b5e84ff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:26:37.184 INFO 4486 --- [host-check-agent-status3,fdc3730f499ab8dd,4a306fbc9b5e84ff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:26:37.184 INFO 4486 --- [host-check-agent-status3,fdc3730f499ab8dd,4a306fbc9b5e84ff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:26:37.184 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:26:37.184 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-agent-status1,fdc3730f499ab8dd,9cf28b36398d8f1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:26:37.184777042+08:00 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-agent-status1,fdc3730f499ab8dd,9cf28b36398d8f1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-agent-status1,fdc3730f499ab8dd,9cf28b36398d8f1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:26:37.182+08:00 and status AVAILABLE 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-agent-status4,fdc3730f499ab8dd,f65a4d4f37376473] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-agent-status4,fdc3730f499ab8dd,f65a4d4f37376473] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-agent-status4,fdc3730f499ab8dd,f65a4d4f37376473] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:26:37.185 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:26:37.183+08:00 2024-10-16 13:26:37.186 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:26:37.185+08:00 2024-10-16 13:26:37.186 INFO 4486 --- [host-check-agent-status2,fdc3730f499ab8dd,debb4efdd9cbd365] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:26:37.186 INFO 4486 --- [host-check-agent-status2,fdc3730f499ab8dd,debb4efdd9cbd365] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:26:37.186 INFO 4486 --- [host-check-agent-status2,fdc3730f499ab8dd,debb4efdd9cbd365] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:26:37.186 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:26:37.182+08:00 and status AVAILABLE 2024-10-16 13:26:37.187 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:26:37.188 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:26:37.188 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:26:37.183+08:00 and status AVAILABLE 2024-10-16 13:26:37.189 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:26:37.185+08:00 and status AVAILABLE 2024-10-16 13:26:37.190 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.190 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:26:37.191 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:26:37.195 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:26:37.195 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.196 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:26:37.196 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:26:37.196 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.197 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:26:37.197 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:26:37.197 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:26:37.198 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.199 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:26:37.199 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.200 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:37.200 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:26:37.200 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:26:37.202 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:37.202 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.202 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:26:37.203 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:37.207 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.211 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.211 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.211 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.211 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.211 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.211 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 13:26:37.212 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.213 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.214 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:37.215 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.218 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:37.219 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 13:26:37.221 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.222 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:37.222 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:37.224 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.224 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:26:37.224 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:26:37.225 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 13:26:37.227 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.228 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:26:37.229 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:26:37.238 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.239 INFO 4486 --- [ocp-schedules-18,8291a42c0a947c45,a6d7033f9f8c3b80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:26:37.250 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:26:37.258 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:26:37.260 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:26:37.261 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:26:37.271 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:26:37.272 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:26:37.276 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:26:37.276 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:37.285 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:26:37.288 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:37.289 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:37.299 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:26:37.308 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:26:37.321 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:26:37.327 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:26:37.335 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:26:37.338 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:26:37.340 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:37.347 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:26:37.354 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:37.362 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:26:37.365 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:37.367 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:26:37.373 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:26:37.378 INFO 4486 --- [host-check-host-status3,79b25de477156ad0,ec596b06a2a77429] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:26:37.388 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:26:37.392 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:26:37.402 INFO 4486 --- [host-check-host-status4,79b25de477156ad0,81b96d535f717a44] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:26:37.404 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:26:37.415 INFO 4486 --- [host-check-host-status2,79b25de477156ad0,a0eb02ff2bbcb864] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:26:37.443 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:26:37.455 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:37.456 INFO 4486 --- [host-check-host-status1,79b25de477156ad0,75fb78ffa203e902] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:26:37.458 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.459 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:26:37.460 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.460 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.461 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:26:37.474 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.474 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.474 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.474 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.474 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.475 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.504 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:26:37.544 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:26:37.593 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:37.610 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:37.611 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.612 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:26:37.614 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:26:37.620 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.621 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:26:37.622 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.622 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.623 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:26:37.635 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.635 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.635 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.635 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.635 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.635 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.643 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:26:37.849 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:26:37.899 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:37.901 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.902 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:26:37.904 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.904 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.905 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:26:37.916 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.916 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.916 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.916 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.916 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:37.917 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:37.933 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:26:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:26:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:26:37.994 INFO 4486 --- [prometheus-minute-schedule-1,6210d3e7613bef79,d22e1a2286873982] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:37.995 INFO 4486 --- [ocp-schedules-12,7e53b29df7adea43,dcdfd731d14b0980] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 13:26:37.995 INFO 4486 --- [ocp-schedules-12,7e53b29df7adea43,dcdfd731d14b0980] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 13:26:38.001 INFO 4486 --- [prometheus-minute-schedule-1,6210d3e7613bef79,d22e1a2286873982] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:38.023 INFO 4486 --- [pool-overview-container-6,4535c759db65e907,5965466641fc7d37] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.030 INFO 4486 --- [pool-10-thread-5,ce310e8d14ef932e,3fc9ad5a56e9490f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 13:26:38.031 INFO 4486 --- [pool-10-thread-7,ce310e8d14ef932e,c435888892d89f60] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 13:26:38.033 INFO 4486 --- [pool-10-thread-6,ce310e8d14ef932e,162a411fd9d018ae] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 13:26:38.034 INFO 4486 --- [pool-overview-container-6,4535c759db65e907,5965466641fc7d37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.034 INFO 4486 --- [pool-10-thread-8,ce310e8d14ef932e,61ce948aa6b34c81] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 13:26:38.037 INFO 4486 --- [pool-overview-container-6,4535c759db65e907,5965466641fc7d37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:26:38.037 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 13:26:38.037 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 13:26:38.037 INFO 4486 --- [ForkJoinPool-6-worker-11,ce310e8d14ef932e,9352f1186e1ed052] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 13:26:38.037 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 13:26:38.037 INFO 4486 --- [ForkJoinPool-6-worker-11,ce310e8d14ef932e,9352f1186e1ed052] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 13:26:38.037 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 13:26:38.041 INFO 4486 --- [pool-10-thread-6,ce310e8d14ef932e,162a411fd9d018ae] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 13:26:38.041 INFO 4486 --- [pool-10-thread-6,ce310e8d14ef932e,162a411fd9d018ae] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 13:26:38.055 INFO 4486 --- [ForkJoinPool-6-worker-11,ce310e8d14ef932e,9352f1186e1ed052] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.055 INFO 4486 --- [pool-10-thread-5,ce310e8d14ef932e,3fc9ad5a56e9490f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 13:26:38.055 INFO 4486 --- [ForkJoinPool-6-worker-11,ce310e8d14ef932e,9352f1186e1ed052] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:26:38.056 INFO 4486 --- [pool-overview-container-6,4535c759db65e907,5965466641fc7d37] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.057 INFO 4486 --- [ForkJoinPool-6-worker-11,ce310e8d14ef932e,9352f1186e1ed052] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.058 INFO 4486 --- [ForkJoinPool-6-worker-11,ce310e8d14ef932e,9352f1186e1ed052] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:26:38.059 INFO 4486 --- [pool-overview-container-6,4535c759db65e907,5965466641fc7d37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.062 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.062 INFO 4486 --- [pool-overview-container-6,4535c759db65e907,5965466641fc7d37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:26:38.066 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:26:38.078 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.079 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:26:38.081 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.082 WARN 4486 --- [ForkJoinPool-6-worker-11,ce310e8d14ef932e,9352f1186e1ed052] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:26:38.082 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:26:38.082 INFO 4486 --- [pool-10-thread-5,ce310e8d14ef932e,3fc9ad5a56e9490f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 13:26:38.087 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.087 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.088 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:26:38.125 INFO 4486 --- [pool-10-thread-5,ce310e8d14ef932e,3fc9ad5a56e9490f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 13:26:38.126 INFO 4486 --- [ForkJoinPool-6-worker-6,ce310e8d14ef932e,c439c9afa5681072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.128 INFO 4486 --- [ForkJoinPool-6-worker-6,ce310e8d14ef932e,c439c9afa5681072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:26:38.130 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:26:38.131 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.134 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:26:38.140 INFO 4486 --- [ForkJoinPool-6-worker-6,ce310e8d14ef932e,c439c9afa5681072] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 13:26:38.141 INFO 4486 --- [ForkJoinPool-6-worker-6,ce310e8d14ef932e,c439c9afa5681072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.142 INFO 4486 --- [ForkJoinPool-6-worker-6,ce310e8d14ef932e,c439c9afa5681072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:26:38.143 INFO 4486 --- [ForkJoinPool-6-worker-6,ce310e8d14ef932e,c439c9afa5681072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.144 INFO 4486 --- [ForkJoinPool-6-worker-6,ce310e8d14ef932e,c439c9afa5681072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:26:38.150 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.151 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:26:38.151 INFO 4486 --- [ForkJoinPool-6-worker-15,ce310e8d14ef932e,7a1495269d4b804e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.154 INFO 4486 --- [ForkJoinPool-6-worker-15,ce310e8d14ef932e,7a1495269d4b804e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:26:38.157 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.157 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:26:38.158 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:26:38.158 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.160 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:26:38.162 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.166 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.168 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.169 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.169 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:26:38.170 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:26:38.173 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:26:38.174 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.175 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:26:38.184 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.187 INFO 4486 --- [ForkJoinPool-6-worker-15,ce310e8d14ef932e,7a1495269d4b804e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:26:38.189 INFO 4486 --- [ForkJoinPool-6-worker-15,ce310e8d14ef932e,7a1495269d4b804e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.189 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:26:38.191 INFO 4486 --- [ForkJoinPool-6-worker-15,ce310e8d14ef932e,7a1495269d4b804e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:26:38.193 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.194 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.195 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:26:38.196 INFO 4486 --- [ForkJoinPool-6-worker-15,ce310e8d14ef932e,7a1495269d4b804e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.198 INFO 4486 --- [ForkJoinPool-6-worker-15,ce310e8d14ef932e,7a1495269d4b804e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:26:38.199 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:26:38.200 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.203 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:26:38.209 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.210 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:26:38.216 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 13:26:38.216 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.217 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:26:38.222 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.223 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:26:38.225 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:26:38.226 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.227 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 13:26:38.227 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:26:38.229 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:26:38.232 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.233 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:26:38.234 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:26:38.236 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.237 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.238 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.243 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.245 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 13:26:38.245 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:26:38, 2024-10-16 13:26:38] 2024-10-16 13:26:38.247 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:26:38.255 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.257 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.259 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 13:26:38.260 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.262 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:26:38, 2024-10-16 13:26:38] 2024-10-16 13:26:38.262 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:26:38.264 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:26:38.266 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.268 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:26:38, 2024-10-16 13:26:38] 2024-10-16 13:26:38.268 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.270 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:26:38.283 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.286 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.286 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.287 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:26:38.287 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.288 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:38.288 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:26:38.288 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.290 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.290 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:26:38.290 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:26:38.290 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.292 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:26:38.292 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:26:38, 2024-10-16 13:26:38] 2024-10-16 13:26:38.294 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:26:38.294 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.294 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 13:26:38.295 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.297 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 13:26:38.297 INFO 4486 --- [ForkJoinPool-6-worker-2,ce310e8d14ef932e,2a558c6bfa297746] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 13:26:38.297 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:26:38, 2024-10-16 13:26:38] 2024-10-16 13:26:38.298 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:38.298 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.298 INFO 4486 --- [ForkJoinPool-6-worker-4,ce310e8d14ef932e,4acf5b2d25c60df9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:26:38.301 INFO 4486 --- [pool-10-thread-8,ce310e8d14ef932e,61ce948aa6b34c81] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 13:26:38.301 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:38.301 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:38.301 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:38.301 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:38.301 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:26:38.302 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.303 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 13:26:38.327 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:26:38.329 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.330 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:26:38, 2024-10-16 13:26:38] 2024-10-16 13:26:38.334 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 13:26:38.334 INFO 4486 --- [ForkJoinPool-6-worker-13,ce310e8d14ef932e,012348f407459ef2] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 13:26:38.345 INFO 4486 --- [pool-10-thread-7,ce310e8d14ef932e,c435888892d89f60] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 13:26:38.353 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:38.361 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:26:38.382 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:26:38.427 INFO 4486 --- [ocp-schedules-13,829be47aecac0b72,dff0bef3ce66ea68] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:26:38.428 INFO 4486 --- [ocp-schedules-13,829be47aecac0b72,dff0bef3ce66ea68] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:26:38.428 INFO 4486 --- [ocp-schedules-13,829be47aecac0b72,dff0bef3ce66ea68] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:26:38.580 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:26:38.385688, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:26:38.580 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:26:38.385688, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:26:38.618 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:26:38.385688} } 2024-10-16 13:26:38.683 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:26:38.690 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:26:38.693 INFO 4486 --- [ocp-schedules-26,36dac477d2ff1e66,87f09f0902952994] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:26:38.757 INFO 4486 --- [ocp-schedules-3,61d94bca6001507b,0a1c5b30bfa32ef8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:26:38.938 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,032a8e5b2caff356,5c05d7f6a2e4f472] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=032a8e5b2caff356, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:38.938 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,032a8e5b2caff356,5c05d7f6a2e4f472] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=032a8e5b2caff356, duration=1 ms] 2024-10-16 13:26:38.974 INFO 4486 --- [ocp-async-9,9a3dcae0d7f538b7,97f4eef7f6485517] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:39.601 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:40.006 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:26:40.006 INFO 4486 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:26:40.023 INFO 4486 --- [Check obproxy2,bf66bf73c66fbb6a,dcf3c85f03600c83] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:26:40.024 INFO 4486 --- [Check obproxy2,bf66bf73c66fbb6a,dcf3c85f03600c83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:40.026 INFO 4486 --- [Check obproxy1,3a5adadd6e7e5c29,f09040f5950b0947] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:26:40.026 INFO 4486 --- [Check obproxy1,3a5adadd6e7e5c29,f09040f5950b0947] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:40.027 INFO 4486 --- [Collect obproxy1,3dcf31fab8d4dcba,b300f2fede44d0ab] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:26:40.027 INFO 4486 --- [Collect obproxy1,3dcf31fab8d4dcba,b300f2fede44d0ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:40.031 INFO 4486 --- [Collect obproxy2,66f8b10029c2741c,b224d081faac57de] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:26:40.031 INFO 4486 --- [Collect obproxy2,66f8b10029c2741c,b224d081faac57de] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:26:40.048 INFO 4486 --- [Check obproxy2,bf66bf73c66fbb6a,dcf3c85f03600c83] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:26:40.048 INFO 4486 --- [Check obproxy2,bf66bf73c66fbb6a,dcf3c85f03600c83] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:26:40.052 INFO 4486 --- [Check obproxy1,3a5adadd6e7e5c29,f09040f5950b0947] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:26:40.052 INFO 4486 --- [Check obproxy1,3a5adadd6e7e5c29,f09040f5950b0947] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:26:40.056 INFO 4486 --- [Collect obproxy2,66f8b10029c2741c,b224d081faac57de] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:26:40.057 INFO 4486 --- [Collect obproxy1,3dcf31fab8d4dcba,b300f2fede44d0ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:26:40.061 INFO 4486 --- [Collect obproxy1,3dcf31fab8d4dcba,b300f2fede44d0ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:26:40.061 INFO 4486 --- [Collect obproxy2,66f8b10029c2741c,b224d081faac57de] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:26:40.062 INFO 4486 --- [ocp-schedules-23,05d300da46125967,f9a46be764e606ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:26:40.064 INFO 4486 --- [Collect obproxy1,3dcf31fab8d4dcba,b300f2fede44d0ab] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:26:40.066 INFO 4486 --- [Collect obproxy2,66f8b10029c2741c,b224d081faac57de] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:26:40.613 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0531ee1e7f48982c,d0203d42aa88ace9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0531ee1e7f48982c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:40.613 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0531ee1e7f48982c,d0203d42aa88ace9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0531ee1e7f48982c, duration=1 ms] 2024-10-16 13:26:40.974 INFO 4486 --- [ocp-async-3,cfb4c2c9ecb9dfe2,c6dc95b32950e8af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:40.994 INFO 4486 --- [prometheus-minute-schedule-1,676ad34c2b585ccd,824f11c3f39676de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:40.995 INFO 4486 --- [prometheus-hour-schedule-1,3a856c0a7ace6176,f0e2817b2b6f3ecd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:26:41.000 INFO 4486 --- [prometheus-minute-schedule-1,676ad34c2b585ccd,824f11c3f39676de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:41.024 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,bcdfc79450f04bba,7584ea1db0cd5705] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bcdfc79450f04bba, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:41.024 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,bcdfc79450f04bba,7584ea1db0cd5705] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bcdfc79450f04bba, duration=1 ms] 2024-10-16 13:26:41.252 INFO 4486 --- [ocp-schedules-2,a534b18079703e5c,158ca92b7027b139] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:26:41.295 WARN 4486 --- [scanner-13232252-1,a534b18079703e5c,46c4dbf7aa28d55c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:26:41.306 INFO 4486 --- [alarm-detect-11,a534b18079703e5c,ae55abeef0a3da39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:41.311 INFO 4486 --- [alarm-detect-11,a534b18079703e5c,ae55abeef0a3da39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:41.316 INFO 4486 --- [alarm-detect-11,a534b18079703e5c,ae55abeef0a3da39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:26:41.323 INFO 4486 --- [alarm-detect-13,a534b18079703e5c,1889d41814dff85b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:41.324 INFO 4486 --- [alarm-detect-10,a534b18079703e5c,db82a2eebdb86191] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:41.328 INFO 4486 --- [alarm-detect-13,a534b18079703e5c,1889d41814dff85b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:41.328 INFO 4486 --- [alarm-detect-10,a534b18079703e5c,db82a2eebdb86191] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:41.332 INFO 4486 --- [alarm-detect-13,a534b18079703e5c,1889d41814dff85b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:26:41.332 INFO 4486 --- [alarm-detect-10,a534b18079703e5c,db82a2eebdb86191] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:26:41.356 INFO 4486 --- [alarm-detect-11,a534b18079703e5c,20e2d95b0f6e2095] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056375 ~ 1729056391 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056390, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056390, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056390=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:26:41.393 INFO 4486 --- [alarm-detect-12,a534b18079703e5c,d3aa82afff796cf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:41.398 INFO 4486 --- [alarm-detect-12,a534b18079703e5c,d3aa82afff796cf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:41.403 INFO 4486 --- [alarm-detect-12,a534b18079703e5c,d3aa82afff796cf8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:26:42.059 INFO 4486 --- [ocp-schedules-1,c80638f91aa609f0,0ef85e74ab36a236] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:26:42.602 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:42.974 INFO 4486 --- [ocp-async-4,db25fcd82de4b281,b5832409e53688d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:26:42.974 INFO 4486 --- [ocp-async-8,cb173c906295deb2,d3b3f86751049ca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:43.994 INFO 4486 --- [prometheus-minute-schedule-1,a8e99436be6358e6,098a2e82cb682617] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:44.000 INFO 4486 --- [prometheus-minute-schedule-1,a8e99436be6358e6,098a2e82cb682617] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:44.975 INFO 4486 --- [ocp-async-10,bef21a6dcdb71548,30b727b49a43a084] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:45.405 INFO 4486 --- [ocp-schedules-23,984a04f6032116d8,a08f96d48d9b675a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:26:45.410 INFO 4486 --- [ocp-schedules-23,984a04f6032116d8,a08f96d48d9b675a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:26:45.410 INFO 4486 --- [ocp-schedules-23,984a04f6032116d8,a08f96d48d9b675a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:26:45.418 INFO 4486 --- [ocp-schedules-25,5bf7861ff3fdd7b1,129d9a8c6e6854d8] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:26:45.418 INFO 4486 --- [ocp-schedules-25,5bf7861ff3fdd7b1,129d9a8c6e6854d8] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:26:45.604 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:46.783 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,cf01241fb25ea05f,c0cb5be2c5de9abb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cf01241fb25ea05f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:46.784 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,cf01241fb25ea05f,c0cb5be2c5de9abb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cf01241fb25ea05f, duration=2 ms] 2024-10-16 13:26:46.975 INFO 4486 --- [ocp-async-7,a4e85b63aa8d8efc,bdf33c443d2740dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:46.994 INFO 4486 --- [prometheus-minute-schedule-1,fb9973efc35558db,cee68a51b2b18c89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:47.000 INFO 4486 --- [prometheus-minute-schedule-1,fb9973efc35558db,cee68a51b2b18c89] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:47.040 INFO 4486 --- [ocp-schedules-5,33b85f58bba0cd54,e35321bcd776fce3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:26:47.071 INFO 4486 --- [ocp-schedules-10,77da91f06998ce6e,d4c12cfe20457c6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:26:47.185 INFO 4486 --- [ocp-schedules-8,32d4d7720ab342d4,b227d3de4a4a33d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:26:47.228 INFO 4486 --- [ocp-schedules-6,1ef226ea17f7e19b,7af5d25f2b4317d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 13:26:48.502 INFO 4486 --- [ocp-schedules-5,8861c1832598f6cc,3d2f760f2772a703] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:26:48.506 INFO 4486 --- [ocp-schedules-5,8861c1832598f6cc,3d2f760f2772a703] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:48.604 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:48.951 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d5450382d50028d4,08a6296eda330fa2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d5450382d50028d4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:48.951 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d5450382d50028d4,08a6296eda330fa2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d5450382d50028d4, duration=2 ms] 2024-10-16 13:26:48.975 INFO 4486 --- [ocp-async-4,052045d63b9a810e,2ccb8041123dbc45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:49.488 INFO 4486 --- [ocp-schedules-15,d6d0833e46dc2d8d,7ec55375feb6cb20] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:26:49.488 INFO 4486 --- [ocp-schedules-15,d6d0833e46dc2d8d,7ec55375feb6cb20] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:26:49.488 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:26:49.488 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:26:49.489 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:26:49.489 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:26:49.489 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:26:49.494 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:26:49.495 INFO 4486 --- [ocp-schedules-15,d6d0833e46dc2d8d,7ec55375feb6cb20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:26:49.495 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:26:49.495 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:26:49.495 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:26:49.496 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:49.496 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:26:49.496 INFO 4486 --- [ocp-schedules-15,d6d0833e46dc2d8d,7ec55375feb6cb20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:26:49.496 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:26:49.496 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:26:49.497 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:26:49.498 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:49.521 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:26:49.521 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:26:49.522 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:26:49.523 INFO 4486 --- [ocp-schedules-15,d6d0833e46dc2d8d,7ec55375feb6cb20] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:26:49.524 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:26:49.526 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:26:49.541 INFO 4486 --- [ocp-schedules-15,d6d0833e46dc2d8d,7ec55375feb6cb20] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:26:49.994 INFO 4486 --- [prometheus-minute-schedule-1,28f1e42996994fb6,18e675f579d11658] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:49.999 INFO 4486 --- [prometheus-minute-schedule-1,28f1e42996994fb6,18e675f579d11658] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:50.067 INFO 4486 --- [ocp-schedules-12,09a152dd82181d27,8b36bb2f47ed7ff3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:26:50.319 INFO 4486 --- [ocp-schedules-9,db0f81304eac611c,045d5a03a7e93741] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:26:50.319 INFO 4486 --- [ocp-schedules-9,db0f81304eac611c,045d5a03a7e93741] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:26:50.325 INFO 4486 --- [ocp-schedules-9,db0f81304eac611c,045d5a03a7e93741] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:26:50.325 INFO 4486 --- [ocp-schedules-9,db0f81304eac611c,045d5a03a7e93741] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:26:50.329 INFO 4486 --- [ocp-schedules-9,db0f81304eac611c,045d5a03a7e93741] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:26:50.329 INFO 4486 --- [ocp-schedules-9,db0f81304eac611c,045d5a03a7e93741] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703033600770641920 2024-10-16 13:26:50.331 INFO 4486 --- [ocp-schedules-9,db0f81304eac611c,045d5a03a7e93741] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=56, maxSyncedRecordId=703036117311098880 2024-10-16 13:26:50.617 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ddbdd4e6c0f790c5,bf2148d5af6019a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ddbdd4e6c0f790c5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:50.618 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ddbdd4e6c0f790c5,bf2148d5af6019a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ddbdd4e6c0f790c5, duration=2 ms] 2024-10-16 13:26:50.976 INFO 4486 --- [ocp-async-6,472f04499b8f8488,b9a1184f0fe2eeb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:51.003 INFO 4486 --- [ocp-schedules-16,e67947447464ea4b,9d7ad297c5154c4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:26:51.003 INFO 4486 --- [ocp-schedules-16,7a0021cce31bf574,1e636aeab1b7db47] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:26:51.006 INFO 4486 --- [ocp-schedules-16,7a0021cce31bf574,1e636aeab1b7db47] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:26:51.030 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,63760455ebc2159f,5446f5c0c26ab022] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=63760455ebc2159f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:51.031 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,63760455ebc2159f,5446f5c0c26ab022] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=63760455ebc2159f, duration=3 ms] 2024-10-16 13:26:51.032 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.033 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:51.053 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.056 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:51.068 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.070 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.070 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:26:51.079 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.080 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:51.099 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.100 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.100 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:51.101 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:26:51.101 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.105 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:26:51.108 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:51.117 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.118 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:51.130 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.139 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.140 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:51.141 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:26:51.146 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:26:51.150 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:26:51.152 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:26:51.176 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:26:51.191 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.193 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.194 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:26:51.226 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:26:51.226 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:26:51.241 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.243 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.244 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:26:51.247 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.251 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:26:51.254 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:51.256 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:26:51.266 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.267 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.267 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:26:51.278 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:26:51.278 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:26:51.281 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.281 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:51.306 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.306 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:51.330 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.330 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:51.356 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.357 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:26:51.381 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.381 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:26:51.410 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:26:51.421 INFO 4486 --- [ocp-schedules-23,d4418b08bb906324,10e3ea1a89b9e1ad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:26:51.444 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.446 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.447 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:26:51.447 WARN 4486 --- [scanner-13232538-1,d4418b08bb906324,ab9f07629b3411d0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:26:51.451 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:26:51.460 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:26:51.466 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:26:51.469 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:26:51.515 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:26:51.610 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:51.685 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:26:51.715 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:26:51.717 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:26:51.718 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:26:51.745 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:26:51.745 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:26:52.974 INFO 4486 --- [ocp-async-5,47ee2ccea7e6f404,98f6c5af861ff08c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:26:52.976 INFO 4486 --- [ocp-async-7,19ac3578d170b335,7ed1c9c1ef1d5e87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:52.994 INFO 4486 --- [prometheus-minute-schedule-1,4da6d94d6b8dbac1,60fcc5b30cb92a7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:52.999 INFO 4486 --- [prometheus-minute-schedule-1,4da6d94d6b8dbac1,60fcc5b30cb92a7e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:54.611 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:54.976 INFO 4486 --- [ocp-async-8,b4535d8d1925dfd8,61596083184406c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:55.994 INFO 4486 --- [prometheus-minute-schedule-1,fb551d665dbd0fbc,2dee801abd0e6207] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:55.999 INFO 4486 --- [prometheus-minute-schedule-1,fb551d665dbd0fbc,2dee801abd0e6207] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:26:56.788 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,bbd8cd53ecf6979b,55d1e4b96ea8c481] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bbd8cd53ecf6979b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:56.788 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,bbd8cd53ecf6979b,55d1e4b96ea8c481] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bbd8cd53ecf6979b, duration=1 ms] 2024-10-16 13:26:56.976 INFO 4486 --- [ocp-async-10,2f97b09bf153f212,7011a3829413a07f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:57.612 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:26:58.535 INFO 4486 --- [ocp-schedules-5,8861c1832598f6cc,3d2f760f2772a703] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:26:58.761 INFO 4486 --- [ocp-schedules-30,c1f884643768639a,ce71aafcf29c2689] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:26:58.956 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,a86ee2c7ec2d9f63,d864acc3c3de9521] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a86ee2c7ec2d9f63, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:26:58.956 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,a86ee2c7ec2d9f63,d864acc3c3de9521] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a86ee2c7ec2d9f63, duration=2 ms] 2024-10-16 13:26:58.976 INFO 4486 --- [ocp-async-3,d4239a3a439fa884,a36499c987d9783a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:26:58.994 INFO 4486 --- [prometheus-minute-schedule-1,1dc2044a88b78f86,4453b0a7e9f9524f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056420, interval=60 2024-10-16 13:26:59.000 INFO 4486 --- [prometheus-minute-schedule-1,1dc2044a88b78f86,4453b0a7e9f9524f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:00.073 INFO 4486 --- [ocp-schedules-12,b8402687f3680350,653c5d7978e98578] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:27:00.613 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:00.621 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,49dcc7d86fe537e4,1261ed56894e28c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=49dcc7d86fe537e4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:00.621 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,49dcc7d86fe537e4,1261ed56894e28c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=49dcc7d86fe537e4, duration=1 ms] 2024-10-16 13:27:00.698 INFO 4486 --- [ocp-schedules-30,810c71b2c64c334c,57f38755c7a5864a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:27:00.697+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:27:00.702 INFO 4486 --- [ocp-schedules-30,810c71b2c64c334c,57f38755c7a5864a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:27:00.706 INFO 4486 --- [ocp-schedules-30,810c71b2c64c334c,57f38755c7a5864a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:27:00.716 INFO 4486 --- [ocp-schedules-30,810c71b2c64c334c,57f38755c7a5864a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:27:00.977 INFO 4486 --- [ocp-async-2,17fe9c7cff9b84af,9271acaedc517a90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:01.036 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,badbc8d2e56b1b8e,4d567bcaac198332] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=badbc8d2e56b1b8e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:01.036 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,badbc8d2e56b1b8e,4d567bcaac198332] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=badbc8d2e56b1b8e, duration=1 ms] 2024-10-16 13:27:01.661 INFO 4486 --- [ocp-schedules-13,774247fc75f20117,348626fb3d288081] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:27:01.722 WARN 4486 --- [scanner-13232804-1,774247fc75f20117,352a350a3535ff21] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:27:01.773 INFO 4486 --- [alarm-detect-1,774247fc75f20117,10f9adb18660f007] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:01.786 INFO 4486 --- [alarm-detect-1,774247fc75f20117,10f9adb18660f007] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:01.787 INFO 4486 --- [alarm-detect-10,774247fc75f20117,181401c15d23d073] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:01.792 INFO 4486 --- [alarm-detect-1,774247fc75f20117,10f9adb18660f007] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:01.793 INFO 4486 --- [alarm-detect-10,774247fc75f20117,181401c15d23d073] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:01.798 INFO 4486 --- [alarm-detect-10,774247fc75f20117,181401c15d23d073] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:01.818 INFO 4486 --- [alarm-detect-16,774247fc75f20117,3dd349fe668147a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:01.819 INFO 4486 --- [alarm-detect-3,774247fc75f20117,b77d04da3c2ad344] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056390 ~ 1729056411 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056405, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056405, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056405=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:27:01.826 INFO 4486 --- [alarm-detect-16,774247fc75f20117,3dd349fe668147a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:01.834 INFO 4486 --- [alarm-detect-16,774247fc75f20117,3dd349fe668147a0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:01.905 INFO 4486 --- [alarm-detect-18,774247fc75f20117,e45ead3702cb1866] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:01.914 INFO 4486 --- [alarm-detect-18,774247fc75f20117,e45ead3702cb1866] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:01.924 INFO 4486 --- [alarm-detect-18,774247fc75f20117,e45ead3702cb1866] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:01.994 INFO 4486 --- [prometheus-minute-schedule-1,6daa376c304ba0db,228d639ff9b7154f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:01.999 INFO 4486 --- [prometheus-minute-schedule-1,6daa376c304ba0db,228d639ff9b7154f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:02.974 INFO 4486 --- [ocp-async-4,8f0bf3ad521aca24,4a0e695aaf3ad963] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:27:02.977 INFO 4486 --- [ocp-async-10,4d4ec391e66c704c,eae9ae89802c1f26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:03.614 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:04.977 INFO 4486 --- [ocp-async-3,3ad27955e5695ff9,44bf8e8902570cb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:04.994 INFO 4486 --- [prometheus-minute-schedule-1,eddd4fa8c612cca1,87c4cd2d0c794ca9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:05.001 INFO 4486 --- [prometheus-minute-schedule-1,eddd4fa8c612cca1,87c4cd2d0c794ca9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:06.615 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:06.793 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,4c171b9580519549,bf93246512f7632c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4c171b9580519549, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:06.793 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,4c171b9580519549,bf93246512f7632c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4c171b9580519549, duration=1 ms] 2024-10-16 13:27:06.977 INFO 4486 --- [ocp-async-1,017f8120b7fdc5f1,e4dceaf9ce8e560b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:07.066 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:27:07.067 INFO 4486 --- [ocp-schedules-13,2d02d49d5a170da1,09d5a397da9c9b0b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:27:07.068 INFO 4486 --- [ocp-schedules-13,2d02d49d5a170da1,09d5a397da9c9b0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:27:07.081 INFO 4486 --- [ocp-schedules-22,6a50853e53acde98,14340f7ab73ef113] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:27:07.084 INFO 4486 --- [ocp-schedules-22,6a50853e53acde98,14340f7ab73ef113] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:27:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:27:07.090 INFO 4486 --- [ocp-schedules-3,fa7763df9a8b02fe,c8bcb479ad7b9403] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:27:07.090 INFO 4486 --- [ocp-schedules-10,c6bf950497e319e3,bd38e0768f8c8512] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:27:07.090 INFO 4486 --- [ocp-schedules-27,e9cf07c2dda165da,ae466e3f1e66c125] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:27:07.091 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:27:07.091 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:27:07.091 INFO 4486 --- [host-check-agent-status1,e9cf07c2dda165da,307e83222db42bfb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:27:07.091 INFO 4486 --- [host-check-clockdiff1,c6bf950497e319e3,6e5198a919e575be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:27:07.092 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:27:07.092 INFO 4486 --- [host-check-clockdiff2,c6bf950497e319e3,1f405ab618fe63be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:27:07.093 INFO 4486 --- [host-check-clockdiff3,c6bf950497e319e3,06d3f69c06763f31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:27:07.094 INFO 4486 --- [host-check-clockdiff4,c6bf950497e319e3,7c516b0ce5bb13b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:27:07.094 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:27:07.096 INFO 4486 --- [host-check-agent-status2,e9cf07c2dda165da,3de2f1aa7ab2a71a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:27:07.096 INFO 4486 --- [host-check-agent-status3,e9cf07c2dda165da,1dd4e1f4a5597ea9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:27:07.097 INFO 4486 --- [host-check-agent-status4,e9cf07c2dda165da,23208abb9792dc55] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:27:07.099 INFO 4486 --- [host-check-agent-status1,e9cf07c2dda165da,307e83222db42bfb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:07.100 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:07.101 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:07.101 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:07.102 INFO 4486 --- [host-check-agent-status1,e9cf07c2dda165da,307e83222db42bfb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:07.102 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:07.103 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:07.117 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:07.117 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:07.117 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:07.137 INFO 4486 --- [host-check-agent-status3,e9cf07c2dda165da,1dd4e1f4a5597ea9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:07.137 INFO 4486 --- [host-check-clockdiff1,c6bf950497e319e3,6e5198a919e575be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056427 0 0 , error= 2024-10-16 13:27:07.141 INFO 4486 --- [host-check-clockdiff4,c6bf950497e319e3,7c516b0ce5bb13b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056427 3 3 , error= 2024-10-16 13:27:07.145 INFO 4486 --- [host-check-agent-status4,e9cf07c2dda165da,23208abb9792dc55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:07.146 INFO 4486 --- [host-check-agent-status3,e9cf07c2dda165da,1dd4e1f4a5597ea9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:07.151 INFO 4486 --- [host-check-clockdiff3,c6bf950497e319e3,06d3f69c06763f31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056427 3 3 , error= 2024-10-16 13:27:07.152 INFO 4486 --- [host-check-agent-status4,e9cf07c2dda165da,23208abb9792dc55] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:07.152 INFO 4486 --- [host-check-agent-status2,e9cf07c2dda165da,3de2f1aa7ab2a71a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:07.154 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.154 INFO 4486 --- [host-check-agent-status2,e9cf07c2dda165da,3de2f1aa7ab2a71a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:07.155 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:07.157 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:27:07.173 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:27:07.173 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:27:07.175 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:27:07.175 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.176 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:27:07.176 INFO 4486 --- [host-check-agent-status1,e9cf07c2dda165da,307e83222db42bfb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:27:07.177 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:07.177 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:27:07.178857928+08:00 2024-10-16 13:27:07.177 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:27:07.179346375+08:00 2024-10-16 13:27:07.178 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:27:07.177+08:00 2024-10-16 13:27:07.178 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:27:07.177+08:00 2024-10-16 13:27:07.178 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:27:07.180678736+08:00 2024-10-16 13:27:07.179 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:27:07.178+08:00 2024-10-16 13:27:07.179 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:27:07.178017219+08:00 2024-10-16 13:27:07.179 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:27:07.179+08:00 2024-10-16 13:27:07.181 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:27:07.177+08:00 and status AVAILABLE 2024-10-16 13:27:07.181 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:27:07.181 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:27:07.177+08:00 and status AVAILABLE 2024-10-16 13:27:07.181 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:27:07.178+08:00 and status AVAILABLE 2024-10-16 13:27:07.182 INFO 4486 --- [host-check-agent-status3,e9cf07c2dda165da,1dd4e1f4a5597ea9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:27:07.182 INFO 4486 --- [host-check-agent-status1,e9cf07c2dda165da,307e83222db42bfb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:27:07.182 INFO 4486 --- [host-check-agent-status1,e9cf07c2dda165da,307e83222db42bfb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:27:07.182 INFO 4486 --- [host-check-agent-status1,e9cf07c2dda165da,307e83222db42bfb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:27:07.182 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:27:07.179+08:00 and status AVAILABLE 2024-10-16 13:27:07.183 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:27:07.183 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:27:07.183 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:27:07.183 INFO 4486 --- [host-check-agent-status2,e9cf07c2dda165da,3de2f1aa7ab2a71a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:27:07.185 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:27:07.185 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.185 INFO 4486 --- [host-check-agent-status3,e9cf07c2dda165da,1dd4e1f4a5597ea9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:27:07.185 INFO 4486 --- [host-check-agent-status3,e9cf07c2dda165da,1dd4e1f4a5597ea9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:27:07.185 INFO 4486 --- [host-check-agent-status3,e9cf07c2dda165da,1dd4e1f4a5597ea9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:27:07.186 INFO 4486 --- [host-check-agent-status4,e9cf07c2dda165da,23208abb9792dc55] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:27:07.188 INFO 4486 --- [host-check-agent-status2,e9cf07c2dda165da,3de2f1aa7ab2a71a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:27:07.188 INFO 4486 --- [host-check-agent-status2,e9cf07c2dda165da,3de2f1aa7ab2a71a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:27:07.188 INFO 4486 --- [host-check-agent-status2,e9cf07c2dda165da,3de2f1aa7ab2a71a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:27:07.190 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:27:07.190 INFO 4486 --- [host-check-agent-status4,e9cf07c2dda165da,23208abb9792dc55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:27:07.190 INFO 4486 --- [host-check-agent-status4,e9cf07c2dda165da,23208abb9792dc55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:27:07.190 INFO 4486 --- [host-check-agent-status4,e9cf07c2dda165da,23208abb9792dc55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:27:07.192 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:07.193 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:07.193 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:07.194 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:07.194 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:07.195 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:07.196 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:07.197 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.198 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:07.198 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:27:07.209 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.210 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:27:07.218 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:07.218 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:07.220 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:07.220 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.221 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:27:07.222 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:07.223 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.225 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:27:07.237 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:07.238 INFO 4486 --- [ocp-schedules-9,32118cf2f3a5d5e1,ca4e031a95ae2c28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:27:07.247 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:27:07.251 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:27:07.260 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:27:07.261 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:27:07.264 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:27:07.271 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:27:07.275 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:07.282 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:07.282 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:27:07.287 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:07.292 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:27:07.303 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:27:07.306 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:07.313 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:27:07.314 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:27:07.326 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:27:07.328 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:07.335 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:27:07.345 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:27:07.345 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:07.359 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:27:07.360 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:07.370 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:27:07.372 INFO 4486 --- [host-check-host-status3,fa7763df9a8b02fe,fd5222a32d94bd34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:27:07.379 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:27:07.382 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:27:07.393 INFO 4486 --- [host-check-host-status4,fa7763df9a8b02fe,655e41f5b17a355b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:27:07.397 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:27:07.399 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:07.412 INFO 4486 --- [host-check-host-status2,fa7763df9a8b02fe,fdd29179e343de41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:27:07.452 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:27:07.463 INFO 4486 --- [host-check-host-status1,fa7763df9a8b02fe,cc07e5baacb4dcf0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:27:07.994 INFO 4486 --- [prometheus-minute-schedule-1,e7af60a1d147d7eb,315819898c4ed5e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:08.001 INFO 4486 --- [prometheus-minute-schedule-1,e7af60a1d147d7eb,315819898c4ed5e4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:08.069 INFO 4486 --- [host-check-clockdiff2,c6bf950497e319e3,1f405ab618fe63be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056428 3 3 , error= 2024-10-16 13:27:08.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e14d687acf8e3a1e,4d030355fd8473b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e14d687acf8e3a1e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:08.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e14d687acf8e3a1e,4d030355fd8473b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e14d687acf8e3a1e, duration=2 ms] 2024-10-16 13:27:08.977 INFO 4486 --- [ocp-async-9,fc0f15ea1714b7c2,d2ef12c46eb44a55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:09.616 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:10.006 INFO 4486 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:27:10.006 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:27:10.026 INFO 4486 --- [Check obproxy1,95a2b67eb411eb26,078eea52b072c1e1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:27:10.026 INFO 4486 --- [Check obproxy2,69c0780d259b251a,ab44643aec103deb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:27:10.026 INFO 4486 --- [Check obproxy1,95a2b67eb411eb26,078eea52b072c1e1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:10.026 INFO 4486 --- [Check obproxy2,69c0780d259b251a,ab44643aec103deb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:10.032 INFO 4486 --- [Collect obproxy2,e3e717568154714a,efbf59202672838d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:27:10.033 INFO 4486 --- [Collect obproxy2,e3e717568154714a,efbf59202672838d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:10.033 INFO 4486 --- [Collect obproxy1,04893d04c9068fae,d82f6ec802cd5443] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:27:10.033 INFO 4486 --- [Collect obproxy1,04893d04c9068fae,d82f6ec802cd5443] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:10.050 INFO 4486 --- [Check obproxy2,69c0780d259b251a,ab44643aec103deb] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:27:10.050 INFO 4486 --- [Check obproxy2,69c0780d259b251a,ab44643aec103deb] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:27:10.056 INFO 4486 --- [Collect obproxy2,e3e717568154714a,efbf59202672838d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:27:10.059 INFO 4486 --- [Check obproxy1,95a2b67eb411eb26,078eea52b072c1e1] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:27:10.059 INFO 4486 --- [Check obproxy1,95a2b67eb411eb26,078eea52b072c1e1] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:27:10.061 INFO 4486 --- [Collect obproxy2,e3e717568154714a,efbf59202672838d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:27:10.066 INFO 4486 --- [Collect obproxy2,e3e717568154714a,efbf59202672838d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:27:10.067 INFO 4486 --- [Collect obproxy1,04893d04c9068fae,d82f6ec802cd5443] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:27:10.071 INFO 4486 --- [Collect obproxy1,04893d04c9068fae,d82f6ec802cd5443] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:27:10.074 INFO 4486 --- [Collect obproxy1,04893d04c9068fae,d82f6ec802cd5443] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:27:10.078 INFO 4486 --- [ocp-schedules-9,cf022807b6319599,5d79cb37a4e1f122] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:27:10.625 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,80efa3d2a31343cf,8742ae499682b332] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=80efa3d2a31343cf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:10.626 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,80efa3d2a31343cf,8742ae499682b332] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=80efa3d2a31343cf, duration=2 ms] 2024-10-16 13:27:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:27:10.978 INFO 4486 --- [ocp-async-7,c0d6219a30f92e6e,fc45a263e17380f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:10.994 INFO 4486 --- [prometheus-minute-schedule-1,afddd4aaaf354b06,f859334d90f12cc4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:11.002 INFO 4486 --- [prometheus-minute-schedule-1,afddd4aaaf354b06,f859334d90f12cc4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:27:11.040 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,06a566e32e91dca4,ecf5f609b4f03288] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=06a566e32e91dca4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:11.040 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,06a566e32e91dca4,ecf5f609b4f03288] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=06a566e32e91dca4, duration=1 ms] 2024-10-16 13:27:11.633 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5c6c844e537449d8,f536d8fcf3e67ed0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=5c6c844e537449d8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:27:11.634 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5c6c844e537449d8,f536d8fcf3e67ed0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=5c6c844e537449d8, duration=2 ms] 2024-10-16 13:27:11.990 INFO 4486 --- [ocp-schedules-1,8a3998ce06551b15,3a2e0ba9a72e62da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:27:12.035 WARN 4486 --- [scanner-13233612-1,8a3998ce06551b15,bdf6871e81b50482] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:27:12.115 INFO 4486 --- [alarm-detect-13,8a3998ce06551b15,b3447c5f9f945483] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056405 ~ 1729056422 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056420, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056420, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056420=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:27:12.617 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:12.857 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056360, nodeCount=274, affectRows=274 2024-10-16 13:27:12.858 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056360, nodeCount=273, validCount=273 2024-10-16 13:27:12.860 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056360, nodeCount=335, affectRows=335 2024-10-16 13:27:12.861 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056360, nodeCount=335, validCount=335 2024-10-16 13:27:12.866 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056360, nodeCount=466, affectRows=466 2024-10-16 13:27:12.869 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056360, nodeCount=466, validCount=466 2024-10-16 13:27:12.870 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056360, nodeCount=502, affectRows=502 2024-10-16 13:27:12.872 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056360, nodeCount=501, validCount=501 2024-10-16 13:27:12.878 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056360, nodeCount=1000, affectRows=1000 2024-10-16 13:27:12.882 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056360, nodeCount=999, validCount=999 2024-10-16 13:27:12.888 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056360, nodeCount=929, affectRows=929 2024-10-16 13:27:12.892 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056360, nodeCount=929, validCount=929 2024-10-16 13:27:12.975 INFO 4486 --- [ocp-async-5,d36578e1f47590ad,5b1870f26c1acc46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:27:12.978 INFO 4486 --- [ocp-async-2,dc12b3807ffeab6c,815360ba190d55a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:13.994 INFO 4486 --- [prometheus-minute-schedule-1,02871f948cc9da86,629507b468531f55] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:14.001 INFO 4486 --- [prometheus-minute-schedule-1,02871f948cc9da86,629507b468531f55] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:14.263 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056360, nodeCount=480, affectRows=480 2024-10-16 13:27:14.902 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056360, nodeCount=480, validCount=480 2024-10-16 13:27:14.978 INFO 4486 --- [ocp-async-4,7f8e0c3cbeb38e47,6999e56278a37737] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:15.298 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,00e536dadbd4086c,ed964ebea60b7005] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=00e536dadbd4086c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:27:15.299 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,00e536dadbd4086c,ed964ebea60b7005] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=00e536dadbd4086c, duration=3 ms] 2024-10-16 13:27:15.618 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:16.797 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,68a43f6931e8e3d1,78a4068fa8367d86] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=68a43f6931e8e3d1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:16.797 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,68a43f6931e8e3d1,78a4068fa8367d86] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=68a43f6931e8e3d1, duration=1 ms] 2024-10-16 13:27:16.978 INFO 4486 --- [ocp-async-7,b0e6f986668365ba,431382ea98705a8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:16.994 INFO 4486 --- [prometheus-minute-schedule-1,62bf6b7e1c845802,d8ff488dca1c310f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:16.999 INFO 4486 --- [prometheus-minute-schedule-1,62bf6b7e1c845802,d8ff488dca1c310f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:17.043 INFO 4486 --- [ocp-schedules-17,ae38a7044c24dbcc,2843f5b88d5aeb8e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:27:17.071 INFO 4486 --- [ocp-schedules-13,2d02d49d5a170da1,09d5a397da9c9b0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:27:17.181 INFO 4486 --- [ocp-schedules-22,6a50853e53acde98,14340f7ab73ef113] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:27:18.619 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:18.766 INFO 4486 --- [ocp-schedules-2,0dd7fe0dfdc44228,e5cb50dc25349163] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:27:18.969 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,daa7ebc7bd85f44f,ebe751a05d5b98e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=daa7ebc7bd85f44f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:18.970 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,daa7ebc7bd85f44f,ebe751a05d5b98e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=daa7ebc7bd85f44f, duration=2 ms] 2024-10-16 13:27:18.978 INFO 4486 --- [ocp-async-1,2c178dc437f42107,f8a03609b4bb34f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:19.994 INFO 4486 --- [prometheus-minute-schedule-1,40e6e426505122cf,fe37405eeba69048] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:20.000 INFO 4486 --- [prometheus-minute-schedule-1,40e6e426505122cf,fe37405eeba69048] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:20.083 INFO 4486 --- [ocp-schedules-27,8227f0224b743342,f4dd57ed0f958b61] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:27:20.630 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1a920f81f0544fe1,1f91375869beaaf9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1a920f81f0544fe1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:20.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1a920f81f0544fe1,1f91375869beaaf9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1a920f81f0544fe1, duration=2 ms] 2024-10-16 13:27:20.868 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 13:27:20.868 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 13:27:20.979 INFO 4486 --- [ocp-async-9,94c79ebf06ad5f64,515e484cc0804646] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:20.996 INFO 4486 --- [prometheus-cache-monitor1,ebb458ab736c7764,935157ae70c14a79] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:27:20.996 INFO 4486 --- [prometheus-cache-monitor1,ebb458ab736c7764,935157ae70c14a79] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:27:21.045 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,49bb98e8a8aba933,6459d521e34d8e50] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=49bb98e8a8aba933, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:21.045 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,49bb98e8a8aba933,6459d521e34d8e50] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=49bb98e8a8aba933, duration=1 ms] 2024-10-16 13:27:21.620 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:21.751 INFO 4486 --- [ocp-schedules-16,7a0021cce31bf574,1e636aeab1b7db47] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:27:21.751 INFO 4486 --- [ocp-schedules-16,d5064ac37b49bffa,e13d7a552ccf51f0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:27:21.754 INFO 4486 --- [ocp-schedules-16,d5064ac37b49bffa,e13d7a552ccf51f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:27:21.781 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.782 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:21.793 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.794 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:21.814 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:21.815 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.815 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:27:21.816 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.817 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:21.844 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:21.845 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.845 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:21.846 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:27:21.847 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.850 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:27:21.851 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:21.860 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:21.860 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:21.870 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:21.881 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:21.882 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.883 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:21.884 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:27:21.891 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:27:21.897 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:21.899 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:21.926 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:21.927 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.928 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:27:21.929 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:27:21.965 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:27:21.965 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:27:21.984 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:21.985 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.986 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:21.989 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:21.992 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:27:21.995 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:22.014 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:27:22.019 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:22.019 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:22.024 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:22.025 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:22.026 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:27:22.036 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:27:22.036 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:27:22.046 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:22.046 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:22.072 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:22.072 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:22.096 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:22.096 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:22.127 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:22.128 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:22.159 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:22.187 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:22.189 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:22.190 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:22.194 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:27:22.197 INFO 4486 --- [ocp-schedules-3,1273102166e810d3,84717714e4a8cfc0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:27:22.200 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:27:22.205 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:27:22.207 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:27:22.235 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:27:22.237 WARN 4486 --- [scanner-13234197-1,1273102166e810d3,178afa222d8ff2ec] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:27:22.262 INFO 4486 --- [alarm-detect-12,1273102166e810d3,e474d90db9fc1dbc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:22.268 WARN 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : Failed to get ob process info, errMsg:[AgentClient]:http request is failed, response:Cannot get process info of open /proc/1046526/status: no such file or directory, reason: %!v(MISSING) 2024-10-16 13:27:22.268 INFO 4486 --- [alarm-detect-12,1273102166e810d3,e474d90db9fc1dbc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:22.272 INFO 4486 --- [alarm-detect-12,1273102166e810d3,e474d90db9fc1dbc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:22.273 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 2024-10-16 13:27:22.275 INFO 4486 --- [alarm-detect-11,1273102166e810d3,c69c9671ab04a5c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:22.277 INFO 4486 --- [alarm-detect-20,1273102166e810d3,75d3a2eed7a9b147] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:22.278 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:22.279 INFO 4486 --- [alarm-detect-11,1273102166e810d3,c69c9671ab04a5c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:22.280 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:22.280 INFO 4486 --- [alarm-detect-20,1273102166e810d3,75d3a2eed7a9b147] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:22.283 INFO 4486 --- [alarm-detect-11,1273102166e810d3,c69c9671ab04a5c9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:22.284 INFO 4486 --- [alarm-detect-20,1273102166e810d3,75d3a2eed7a9b147] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:22.304 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:27:22.359 INFO 4486 --- [alarm-detect-6,1273102166e810d3,5eedf764e28ae438] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:22.362 INFO 4486 --- [alarm-detect-6,1273102166e810d3,5eedf764e28ae438] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:22.367 INFO 4486 --- [alarm-detect-6,1273102166e810d3,5eedf764e28ae438] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:22.461 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 got ObProcessInfoResult(pid=91568, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:27:22.487 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:22.489 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:22.490 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:27:22.512 INFO 4486 --- [ocp-schedules-21,293f1cb4ed579ad4,83400eba7caa47ff] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:27:22.521 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:27:22.522 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:27:22.975 INFO 4486 --- [ocp-async-4,4f2fb7482eb4b46e,1d79104ec8e74633] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:27:22.979 INFO 4486 --- [ocp-async-6,e203208519850c8d,1429e0bfd965ba5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:22.994 INFO 4486 --- [prometheus-minute-schedule-1,e69d50bb50774fe9,9d23365a41300e26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:22.999 INFO 4486 --- [prometheus-minute-schedule-1,e69d50bb50774fe9,9d23365a41300e26] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:23.025 INFO 4486 --- [ocp-schedules-14,aa9aaab9eba74843,0ef0d135cf2702b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:27:23.434 INFO 4486 --- [ocp-schedules-11,f05254437d90c952,7b680273b048046e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:27:23.435 INFO 4486 --- [ocp-schedules-11,f05254437d90c952,7b680273b048046e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:27:23.436 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:27:23.436 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:27:23.436 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:27:23.963 INFO 4486 --- [ocp-schedules-6,a2852e47bd71ad7a,184e5f416c390d20] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:27:24.011 INFO 4486 --- [ocp-schedules-25,c4a615c398a9b59d,c61431578e9955ff] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:27:24.011 INFO 4486 --- [ocp-schedules-25,c4a615c398a9b59d,c61431578e9955ff] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:27:24.045 INFO 4486 --- [Collect all tenants major compactions2,c4a615c398a9b59d,8df288433c339014] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:27:24.045 INFO 4486 --- [Collect all tenants major compactions1,c4a615c398a9b59d,069b33a62a0a52cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:27:24.045 INFO 4486 --- [Collect all tenants major compactions3,c4a615c398a9b59d,6dc363e072722fc2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:27:24.046 INFO 4486 --- [Collect all tenants major compactions4,c4a615c398a9b59d,4bd53a51e4639a64] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:27:24.046 INFO 4486 --- [Collect all tenants major compactions5,c4a615c398a9b59d,bdc731c4e73b23d8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:27:24.057 INFO 4486 --- [Collect all tenants major compactions2,c4a615c398a9b59d,8df288433c339014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:24.057 INFO 4486 --- [Collect all tenants major compactions1,c4a615c398a9b59d,069b33a62a0a52cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:24.058 INFO 4486 --- [Collect all tenants major compactions1,c4a615c398a9b59d,069b33a62a0a52cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:27:24.058 INFO 4486 --- [Collect all tenants major compactions2,c4a615c398a9b59d,8df288433c339014] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:27:24.061 INFO 4486 --- [Collect all tenants major compactions1,c4a615c398a9b59d,069b33a62a0a52cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:27:24.061 INFO 4486 --- [Collect all tenants major compactions2,c4a615c398a9b59d,8df288433c339014] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:27:24.061 INFO 4486 --- [Collect all tenants major compactions1,c4a615c398a9b59d,069b33a62a0a52cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:27:24.062 INFO 4486 --- [Collect all tenants major compactions2,c4a615c398a9b59d,8df288433c339014] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:27:24.064 INFO 4486 --- [Collect all tenants major compactions3,c4a615c398a9b59d,6dc363e072722fc2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:24.065 INFO 4486 --- [Collect all tenants major compactions3,c4a615c398a9b59d,6dc363e072722fc2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:27:24.069 INFO 4486 --- [Collect all tenants major compactions3,c4a615c398a9b59d,6dc363e072722fc2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:27:24.069 INFO 4486 --- [Collect all tenants major compactions3,c4a615c398a9b59d,6dc363e072722fc2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:27:24.077 INFO 4486 --- [Collect all tenants major compactions4,c4a615c398a9b59d,4bd53a51e4639a64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:24.078 INFO 4486 --- [Collect all tenants major compactions5,c4a615c398a9b59d,bdc731c4e73b23d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:24.079 INFO 4486 --- [Collect all tenants major compactions4,c4a615c398a9b59d,4bd53a51e4639a64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:27:24.079 INFO 4486 --- [Collect all tenants major compactions5,c4a615c398a9b59d,bdc731c4e73b23d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:27:24.083 INFO 4486 --- [Collect all tenants major compactions4,c4a615c398a9b59d,4bd53a51e4639a64] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:27:24.083 INFO 4486 --- [Collect all tenants major compactions4,c4a615c398a9b59d,4bd53a51e4639a64] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:27:24.083 INFO 4486 --- [Collect all tenants major compactions5,c4a615c398a9b59d,bdc731c4e73b23d8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:27:24.084 INFO 4486 --- [Collect all tenants major compactions5,c4a615c398a9b59d,bdc731c4e73b23d8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:27:24.085 INFO 4486 --- [ocp-schedules-25,c4a615c398a9b59d,c61431578e9955ff] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:27:24.621 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:24.979 INFO 4486 --- [ocp-async-5,403c13f373bc5283,198ab9e6a5e9b149] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:25.994 INFO 4486 --- [prometheus-minute-schedule-1,63bb9a8c64ec9174,6defecc16f5318d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:26.000 INFO 4486 --- [prometheus-minute-schedule-1,63bb9a8c64ec9174,6defecc16f5318d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:26.636 INFO 4486 --- [ocp-schedules-13,bc55f4bdc528b518,abe9c5540159d106] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:27:26.636+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:27:26.638 INFO 4486 --- [ocp-schedules-13,bc55f4bdc528b518,abe9c5540159d106] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:27:26.639 INFO 4486 --- [ocp-schedules-13,bc55f4bdc528b518,abe9c5540159d106] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:27:26.640 INFO 4486 --- [ocp-schedules-13,bc55f4bdc528b518,abe9c5540159d106] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:27:26.808 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0d852615af1e95a5,9897e33da2e42e2a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0d852615af1e95a5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:26.808 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0d852615af1e95a5,9897e33da2e42e2a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0d852615af1e95a5, duration=2 ms] 2024-10-16 13:27:26.979 INFO 4486 --- [ocp-async-2,4713ab9c60f1df1f,05ba4cd3c6e87206] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:27.622 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:28.974 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,46587d9e095d3e81,dbf9821ed09e4644] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=46587d9e095d3e81, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:28.974 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,46587d9e095d3e81,dbf9821ed09e4644] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=46587d9e095d3e81, duration=1 ms] 2024-10-16 13:27:28.979 INFO 4486 --- [ocp-async-10,875c2818637e40da,1f2c3fcf9b17bd6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:28.994 INFO 4486 --- [prometheus-minute-schedule-1,fefd6330d4da352f,e4bdbb7715d65c87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:28.999 INFO 4486 --- [prometheus-minute-schedule-1,fefd6330d4da352f,e4bdbb7715d65c87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:29.511 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,a657546426a8e334,58152d170c4adfb9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=a657546426a8e334, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:27:29.512 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,a657546426a8e334,58152d170c4adfb9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=a657546426a8e334, duration=2 ms] 2024-10-16 13:27:30.089 INFO 4486 --- [ocp-schedules-19,8c16b97f53dd5ebe,b57d3c492ca7b513] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:27:30.103 INFO 4486 --- [prometheus-second-schedule-1,c45a00b084c78132,94b490b7ce56362c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:27:30.623 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:30.634 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,320a0af1d96ec9f3,f609e6b49bbe8abd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=320a0af1d96ec9f3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:30.635 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,320a0af1d96ec9f3,f609e6b49bbe8abd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=320a0af1d96ec9f3, duration=1 ms] 2024-10-16 13:27:30.979 INFO 4486 --- [ocp-async-6,a25ea17c1ca634e3,16ce348e89798ed9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:31.049 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,d90a65ed0f2d69cc,b0e2321b1cdd5067] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d90a65ed0f2d69cc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:31.049 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,d90a65ed0f2d69cc,b0e2321b1cdd5067] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d90a65ed0f2d69cc, duration=1 ms] 2024-10-16 13:27:31.994 INFO 4486 --- [prometheus-minute-schedule-1,632287876e6e6f56,3505efc449f010ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:32.000 INFO 4486 --- [prometheus-minute-schedule-1,632287876e6e6f56,3505efc449f010ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:32.398 INFO 4486 --- [ocp-schedules-12,cd12527fa0318b46,ae98f24ae77423e3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:27:32.424 WARN 4486 --- [scanner-13234531-1,cd12527fa0318b46,3e8517f4cb15e7ab] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:27:32.504 INFO 4486 --- [alarm-detect-19,cd12527fa0318b46,759169046f7d865c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056420 ~ 1729056442 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056435, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056435, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056435=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:27:32.975 INFO 4486 --- [ocp-async-5,27d61ec2333706e7,59b6171198b97823] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:27:32.980 INFO 4486 --- [ocp-async-8,6b37b725cdb874b1,42fa68ce42ddb319] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:33.436 INFO 4486 --- [ocp-schedules-11,f05254437d90c952,7b680273b048046e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:27:33.624 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:34.980 INFO 4486 --- [ocp-async-7,e3084d7b7f4cf679,1bc26e02ef90c7ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:34.994 INFO 4486 --- [prometheus-minute-schedule-1,145fa65f98ac0455,0fb515728ae1db7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:35.003 INFO 4486 --- [prometheus-minute-schedule-1,145fa65f98ac0455,0fb515728ae1db7e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:36.625 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:36.814 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,d24b9bf973a93c28,923b580afc5b6676] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d24b9bf973a93c28, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:36.815 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,d24b9bf973a93c28,923b580afc5b6676] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d24b9bf973a93c28, duration=3 ms] 2024-10-16 13:27:36.980 INFO 4486 --- [ocp-async-3,31d7beeb47e05670,a7c19329b0a82fe5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:37.063 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:27:37.064 INFO 4486 --- [ocp-schedules-12,9e2c02a493e94d86,3b62c9d1a417ab18] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:27:37.066 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:27:37.067 INFO 4486 --- [ocp-schedules-11,6c64149690543954,4d555666911797c4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:27:37.068 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:27:37.068 INFO 4486 --- [ocp-schedules-11,6c64149690543954,4d555666911797c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:27:37.070 INFO 4486 --- [ocp-schedules-12,9e2c02a493e94d86,3b62c9d1a417ab18] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:27:37.081 INFO 4486 --- [ocp-schedules-20,18db66aedb5e8494,eb412e6294425cc5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:27:37.084 INFO 4486 --- [ocp-schedules-20,18db66aedb5e8494,eb412e6294425cc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:27:37.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:27:37.090 INFO 4486 --- [ocp-schedules-13,0b013e48d52b0cd7,73111835bbb86b65] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:27:37.090 INFO 4486 --- [ocp-schedules-17,dba8577014a9e4fb,88462147bc86abc6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:27:37.090 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:27:37.090 INFO 4486 --- [host-check-clockdiff1,0b013e48d52b0cd7,7efd1130affabaa5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:27:37.091 INFO 4486 --- [ocp-schedules-24,0cb2af290a834200,630ad9b38f85fd6c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:27:37.091 INFO 4486 --- [host-check-clockdiff2,0b013e48d52b0cd7,00fed9077d7c7243] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:27:37.091 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:27:37.092 INFO 4486 --- [host-check-clockdiff3,0b013e48d52b0cd7,f82705fc55ac2b45] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:27:37.092 INFO 4486 --- [host-check-agent-status1,0cb2af290a834200,cf32b893ed113f68] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:27:37.093 INFO 4486 --- [host-check-clockdiff4,0b013e48d52b0cd7,ab02e9d4b2fea460] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:27:37.094 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:27:37.094 INFO 4486 --- [host-check-agent-status2,0cb2af290a834200,073e12a5677e3b4e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:27:37.094 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:27:37.095 INFO 4486 --- [host-check-agent-status3,0cb2af290a834200,816ad10634ac18bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:27:37.096 INFO 4486 --- [host-check-agent-status4,0cb2af290a834200,063565c459b50768] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:27:37.099 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:37.101 INFO 4486 --- [host-check-agent-status2,0cb2af290a834200,073e12a5677e3b4e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:37.101 INFO 4486 --- [host-check-agent-status1,0cb2af290a834200,cf32b893ed113f68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:37.101 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:37.101 INFO 4486 --- [host-check-agent-status3,0cb2af290a834200,816ad10634ac18bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:37.102 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:37.103 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:37.104 INFO 4486 --- [host-check-agent-status2,0cb2af290a834200,073e12a5677e3b4e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:37.104 INFO 4486 --- [host-check-agent-status1,0cb2af290a834200,cf32b893ed113f68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:37.104 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:37.104 INFO 4486 --- [host-check-agent-status3,0cb2af290a834200,816ad10634ac18bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:37.104 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:37.104 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:37.114 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:37.114 INFO 4486 --- [host-check-agent-status4,0cb2af290a834200,063565c459b50768] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:37.121 INFO 4486 --- [host-check-clockdiff3,0b013e48d52b0cd7,f82705fc55ac2b45] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056457 3 3 , error= 2024-10-16 13:27:37.121 INFO 4486 --- [host-check-clockdiff1,0b013e48d52b0cd7,7efd1130affabaa5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056457 0 0 , error= 2024-10-16 13:27:37.121 INFO 4486 --- [host-check-clockdiff2,0b013e48d52b0cd7,00fed9077d7c7243] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056457 3 3 , error= 2024-10-16 13:27:37.134 INFO 4486 --- [host-check-agent-status4,0cb2af290a834200,063565c459b50768] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:37.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:37.142 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:27:37.154 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:37.156 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.157 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:27:37.158 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.158 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:37.158 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.159 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:37.159 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.159 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.160 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:27:37.161 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:27:37.161 INFO 4486 --- [host-check-agent-status2,0cb2af290a834200,073e12a5677e3b4e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:27:37.161 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:27:37.162 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:27:37.162 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:27:37.163 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:27:37.163 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:27:37.164 INFO 4486 --- [host-check-agent-status1,0cb2af290a834200,cf32b893ed113f68] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:27:37.164 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:27:37.166887411+08:00 2024-10-16 13:27:37.164 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:27:37.166824003+08:00 2024-10-16 13:27:37.164 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:27:37.166723771+08:00 2024-10-16 13:27:37.165 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:27:37.164+08:00 2024-10-16 13:27:37.165 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:27:37.164612356+08:00 2024-10-16 13:27:37.165 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:27:37.164+08:00 2024-10-16 13:27:37.165 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:27:37.164+08:00 2024-10-16 13:27:37.165 INFO 4486 --- [host-check-agent-status3,0cb2af290a834200,816ad10634ac18bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:27:37.165 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:27:37.165+08:00 2024-10-16 13:27:37.167 INFO 4486 --- [host-check-agent-status2,0cb2af290a834200,073e12a5677e3b4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:27:37.167 INFO 4486 --- [host-check-agent-status2,0cb2af290a834200,073e12a5677e3b4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:27:37.167 INFO 4486 --- [host-check-agent-status2,0cb2af290a834200,073e12a5677e3b4e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:27:37.167 INFO 4486 --- [host-check-agent-status4,0cb2af290a834200,063565c459b50768] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:27:37.167 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:27:37.164+08:00 and status AVAILABLE 2024-10-16 13:27:37.167 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:27:37.164+08:00 and status AVAILABLE 2024-10-16 13:27:37.167 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:27:37.164+08:00 and status AVAILABLE 2024-10-16 13:27:37.168 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:27:37.165+08:00 and status AVAILABLE 2024-10-16 13:27:37.168 INFO 4486 --- [host-check-agent-status3,0cb2af290a834200,816ad10634ac18bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:27:37.168 INFO 4486 --- [host-check-agent-status3,0cb2af290a834200,816ad10634ac18bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:27:37.168 INFO 4486 --- [host-check-agent-status3,0cb2af290a834200,816ad10634ac18bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:27:37.169 INFO 4486 --- [host-check-agent-status1,0cb2af290a834200,cf32b893ed113f68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:27:37.169 INFO 4486 --- [host-check-agent-status1,0cb2af290a834200,cf32b893ed113f68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:27:37.169 INFO 4486 --- [host-check-agent-status1,0cb2af290a834200,cf32b893ed113f68] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:27:37.170 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:27:37.170 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:27:37.170 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:27:37.170 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:27:37.170 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.171 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:27:37.172 INFO 4486 --- [host-check-agent-status4,0cb2af290a834200,063565c459b50768] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:27:37.172 INFO 4486 --- [host-check-agent-status4,0cb2af290a834200,063565c459b50768] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:27:37.172 INFO 4486 --- [host-check-agent-status4,0cb2af290a834200,063565c459b50768] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:27:37.173 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.173 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.173 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.173 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.173 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.174 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.180 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:37.180 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:37.181 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:37.181 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:37.182 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:37.182 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:37.182 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:37.182 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:37.183 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.184 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:27:37.193 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.194 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:27:37.195 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:27:37.195 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.197 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:27:37.203 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:37.204 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:37.204 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:37.208 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:27:37.209 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.211 INFO 4486 --- [ocp-schedules-26,ff9ab4e08f4ee425,cb206de3beba20f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:27:37.240 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:27:37.243 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:27:37.249 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:27:37.251 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:27:37.257 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:27:37.261 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:27:37.263 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:27:37.271 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:37.273 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:37.273 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:27:37.277 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:37.288 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:27:37.299 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:27:37.309 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:27:37.312 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:27:37.325 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:27:37.327 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:27:37.330 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:37.340 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:27:37.342 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:27:37.342 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:37.354 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:27:37.357 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:37.358 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:27:37.368 INFO 4486 --- [host-check-host-status3,dba8577014a9e4fb,185b048621e597d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:27:37.370 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:27:37.375 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:27:37.384 INFO 4486 --- [host-check-host-status4,dba8577014a9e4fb,35816a86a4fdffef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:27:37.396 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:27:37.405 INFO 4486 --- [host-check-host-status2,dba8577014a9e4fb,098838b3294a5380] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:27:37.415 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:37.418 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.419 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:27:37.420 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.421 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.422 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:27:37.425 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:27:37.431 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.431 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.431 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.431 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.431 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.432 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.435 INFO 4486 --- [host-check-host-status1,dba8577014a9e4fb,182b9594c66fecbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:27:37.460 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:27:37.506 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:27:37.552 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:37.571 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:37.573 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.573 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:27:37.575 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:27:37.576 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.577 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:27:37.578 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.579 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.582 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:27:37.594 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.594 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.594 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.594 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.594 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.594 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.602 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:27:37.805 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:27:37.852 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:37.854 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.854 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:27:37.855 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.856 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.857 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:27:37.866 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.866 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.866 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.866 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.866 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:37.866 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:37.884 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:27:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:27:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:27:37.994 INFO 4486 --- [prometheus-minute-schedule-1,6278e88c220ea2d9,7b33d89881ac36bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:38.001 INFO 4486 --- [prometheus-minute-schedule-1,6278e88c220ea2d9,7b33d89881ac36bf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:38.051 INFO 4486 --- [host-check-clockdiff4,0b013e48d52b0cd7,ab02e9d4b2fea460] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056458 3 3 , error= 2024-10-16 13:27:38.092 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:27:38.139 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:38.142 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:38.142 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:27:38.143 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:38.144 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:38.145 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:27:38.157 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:38.157 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:38.157 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:38.157 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:38.157 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:27:38.157 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:38.222 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:27:38.427 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:27:38.224818, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:27:38.427 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:27:38.224818, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:27:38.463 INFO 4486 --- [ocp-schedules-18,f6ab713e3ae68e50,fc37425aaaeeda71] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:27:38.463 INFO 4486 --- [ocp-schedules-18,f6ab713e3ae68e50,fc37425aaaeeda71] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:27:38.466 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:27:38.224818} } 2024-10-16 13:27:38.527 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:27:38.535 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:27:38.538 INFO 4486 --- [ocp-schedules-1,6a9e0e3282546d1f,275d003de9abb692] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:27:38.771 INFO 4486 --- [ocp-schedules-14,7c95e8b132b5bd42,c2075b2b8d95fca0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:27:38.980 INFO 4486 --- [ocp-async-8,6ecd0dd150a11c96,f455e65fbef00223] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:38.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,eb1a1e4d7d329a8e,448216e535979453] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eb1a1e4d7d329a8e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:38.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,eb1a1e4d7d329a8e,448216e535979453] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eb1a1e4d7d329a8e, duration=1 ms] 2024-10-16 13:27:39.626 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:40.006 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:27:40.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:27:40.029 INFO 4486 --- [Check obproxy2,859e89f828d9b1bf,8c11e49cff3632a8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:27:40.030 INFO 4486 --- [Check obproxy2,859e89f828d9b1bf,8c11e49cff3632a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:40.031 INFO 4486 --- [Check obproxy1,bdfd191ac509a91f,8751116f56cb0ddf] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:27:40.031 INFO 4486 --- [Check obproxy1,bdfd191ac509a91f,8751116f56cb0ddf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:40.033 INFO 4486 --- [Collect obproxy1,73da0a1b1685c517,4dd43ff4f4b116ad] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:27:40.033 INFO 4486 --- [Collect obproxy1,73da0a1b1685c517,4dd43ff4f4b116ad] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:40.038 INFO 4486 --- [Collect obproxy2,7689305af3586819,68a207f3f1f68f39] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:27:40.038 INFO 4486 --- [Collect obproxy2,7689305af3586819,68a207f3f1f68f39] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:27:40.054 INFO 4486 --- [Check obproxy2,859e89f828d9b1bf,8c11e49cff3632a8] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:27:40.054 INFO 4486 --- [Check obproxy2,859e89f828d9b1bf,8c11e49cff3632a8] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:27:40.055 INFO 4486 --- [Check obproxy1,bdfd191ac509a91f,8751116f56cb0ddf] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:27:40.055 INFO 4486 --- [Check obproxy1,bdfd191ac509a91f,8751116f56cb0ddf] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:27:40.057 INFO 4486 --- [Collect obproxy2,7689305af3586819,68a207f3f1f68f39] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:27:40.060 INFO 4486 --- [Collect obproxy1,73da0a1b1685c517,4dd43ff4f4b116ad] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:27:40.063 INFO 4486 --- [Collect obproxy2,7689305af3586819,68a207f3f1f68f39] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:27:40.063 INFO 4486 --- [Collect obproxy1,73da0a1b1685c517,4dd43ff4f4b116ad] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:27:40.066 INFO 4486 --- [Collect obproxy1,73da0a1b1685c517,4dd43ff4f4b116ad] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:27:40.068 INFO 4486 --- [Collect obproxy2,7689305af3586819,68a207f3f1f68f39] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:27:40.096 INFO 4486 --- [ocp-schedules-25,713611185c763f0f,691637c0489be102] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:27:40.644 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,848093b0a742b4e4,12c5885a2b644458] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=848093b0a742b4e4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:40.644 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,848093b0a742b4e4,12c5885a2b644458] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=848093b0a742b4e4, duration=2 ms] 2024-10-16 13:27:40.980 INFO 4486 --- [ocp-async-7,ceca50790f31a100,0d96e7e29811fc9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:40.994 INFO 4486 --- [prometheus-minute-schedule-1,fe0c3bdf4528d51d,ee7f4a2c14fa5d74] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:40.995 INFO 4486 --- [prometheus-hour-schedule-1,dc5f318a64d7e91a,eee92e85d05e061d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:27:41.000 INFO 4486 --- [prometheus-minute-schedule-1,fe0c3bdf4528d51d,ee7f4a2c14fa5d74] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:41.057 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,216cd98af96152df,a72c31b8c774fae5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=216cd98af96152df, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:41.058 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,216cd98af96152df,a72c31b8c774fae5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=216cd98af96152df, duration=1 ms] 2024-10-16 13:27:42.063 INFO 4486 --- [ocp-schedules-22,b5492a94e6add8e8,9efd5d72cbad3b47] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:27:42.575 INFO 4486 --- [ocp-schedules-17,f01a4178cb178bb6,e0f04007cbe86f5b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:27:42.628 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:42.634 WARN 4486 --- [scanner-13234781-1,f01a4178cb178bb6,461dd476c3c3ea97] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:27:42.665 INFO 4486 --- [alarm-detect-9,f01a4178cb178bb6,f58eae88f8542078] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:42.668 INFO 4486 --- [alarm-detect-11,f01a4178cb178bb6,f86955def022f332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:42.669 INFO 4486 --- [alarm-detect-19,f01a4178cb178bb6,7be6bec9609d0103] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:42.670 INFO 4486 --- [alarm-detect-9,f01a4178cb178bb6,f58eae88f8542078] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:42.672 INFO 4486 --- [alarm-detect-11,f01a4178cb178bb6,f86955def022f332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:42.674 INFO 4486 --- [alarm-detect-19,f01a4178cb178bb6,7be6bec9609d0103] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:42.675 INFO 4486 --- [alarm-detect-9,f01a4178cb178bb6,f58eae88f8542078] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:27:42.678 INFO 4486 --- [alarm-detect-11,f01a4178cb178bb6,f86955def022f332] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:27:42.679 INFO 4486 --- [alarm-detect-19,f01a4178cb178bb6,7be6bec9609d0103] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:27:42.696 INFO 4486 --- [alarm-detect-3,f01a4178cb178bb6,f53d776986cea602] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056435 ~ 1729056452 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056450, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056450, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056450=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:27:42.752 INFO 4486 --- [alarm-detect-13,f01a4178cb178bb6,627631f1c234e49c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:42.759 INFO 4486 --- [alarm-detect-13,f01a4178cb178bb6,627631f1c234e49c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:42.763 INFO 4486 --- [alarm-detect-13,f01a4178cb178bb6,627631f1c234e49c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:27:42.975 INFO 4486 --- [ocp-async-3,7f8553ec43aceecb,ece55441420dcb6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:27:42.981 INFO 4486 --- [ocp-async-1,8a715d419252f309,096ff225a1f6ed7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:43.994 INFO 4486 --- [prometheus-minute-schedule-1,5cfb74cd67f0665f,b911d4116ed5ebc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:44.002 INFO 4486 --- [prometheus-minute-schedule-1,5cfb74cd67f0665f,b911d4116ed5ebc1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:44.981 INFO 4486 --- [ocp-async-9,152f091cc5790d35,70963347fe26a364] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:45.410 INFO 4486 --- [ocp-schedules-10,068ba806177ffb60,a42855c9894f4c0b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:27:45.414 INFO 4486 --- [ocp-schedules-10,068ba806177ffb60,a42855c9894f4c0b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:27:45.414 INFO 4486 --- [ocp-schedules-10,068ba806177ffb60,a42855c9894f4c0b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:27:45.423 INFO 4486 --- [ocp-schedules-25,4826588fe25d97c4,b0a8d5862534bb58] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:27:45.423 INFO 4486 --- [ocp-schedules-25,4826588fe25d97c4,b0a8d5862534bb58] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:27:45.629 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:46.820 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5f0aca683141b2a3,b7d8d75ff94c64ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5f0aca683141b2a3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:46.821 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5f0aca683141b2a3,b7d8d75ff94c64ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5f0aca683141b2a3, duration=2 ms] 2024-10-16 13:27:46.981 INFO 4486 --- [ocp-async-7,3e7fe5f77cf00816,459f302b30506c62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:46.994 INFO 4486 --- [prometheus-minute-schedule-1,fc9a07cb7ec4129a,1fa644a929c9cbcf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:47.000 INFO 4486 --- [prometheus-minute-schedule-1,fc9a07cb7ec4129a,1fa644a929c9cbcf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:47.046 INFO 4486 --- [ocp-schedules-29,33340b0df2ad7d4a,068e8254bd76501a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:27:47.070 INFO 4486 --- [ocp-schedules-11,6c64149690543954,4d555666911797c4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:27:47.164 INFO 4486 --- [ocp-schedules-20,18db66aedb5e8494,eb412e6294425cc5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:27:48.630 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:48.760 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:48.760 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:27:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:27:48.763 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:27:48.981 INFO 4486 --- [ocp-async-2,ac6ca0bbe348a59a,c3c8bbc90e42f5a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:48.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b6092ad095ec93ef,9fbf96c91a7aebf6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b6092ad095ec93ef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:48.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b6092ad095ec93ef,9fbf96c91a7aebf6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b6092ad095ec93ef, duration=0 ms] 2024-10-16 13:27:49.549 INFO 4486 --- [ocp-schedules-22,e77b2d40d2f821e3,057a861e848c8b8b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:27:49.549 INFO 4486 --- [ocp-schedules-22,e77b2d40d2f821e3,057a861e848c8b8b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:27:49.549 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:27:49.549 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:27:49.549 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:27:49.549 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:27:49.549 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:27:49.555 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:49.555 INFO 4486 --- [ocp-schedules-22,e77b2d40d2f821e3,057a861e848c8b8b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:27:49.555 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:27:49.556 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:49.556 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:49.556 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:49.557 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:27:49.557 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:49.557 INFO 4486 --- [ocp-schedules-22,e77b2d40d2f821e3,057a861e848c8b8b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:27:49.557 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:49.558 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:49.558 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:49.581 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:27:49.581 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:27:49.581 INFO 4486 --- [ocp-schedules-22,e77b2d40d2f821e3,057a861e848c8b8b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:27:49.581 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:27:49.581 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:27:49.597 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:27:49.600 INFO 4486 --- [ocp-schedules-22,e77b2d40d2f821e3,057a861e848c8b8b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:27:49.994 INFO 4486 --- [prometheus-minute-schedule-1,47ec6060b5546885,0cc9a544aefe2b11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:50.000 INFO 4486 --- [prometheus-minute-schedule-1,47ec6060b5546885,0cc9a544aefe2b11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:50.101 INFO 4486 --- [ocp-schedules-27,5c29547491e02848,85b701d4c039b1c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:27:50.332 INFO 4486 --- [ocp-schedules-9,c820cdcc12b28b01,7892f3100908457e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:27:50.332 INFO 4486 --- [ocp-schedules-9,c820cdcc12b28b01,7892f3100908457e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:27:50.334 INFO 4486 --- [ocp-schedules-9,c820cdcc12b28b01,7892f3100908457e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:27:50.334 INFO 4486 --- [ocp-schedules-9,c820cdcc12b28b01,7892f3100908457e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:27:50.334 INFO 4486 --- [ocp-schedules-9,c820cdcc12b28b01,7892f3100908457e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:27:50.334 INFO 4486 --- [ocp-schedules-9,c820cdcc12b28b01,7892f3100908457e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:27:50.503 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:27:50.503 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:27:50.515 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:27:50.515 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:27:50.515 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:27:50.526 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:27:50.526 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:27:50.526 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:27:50.531 INFO 4486 --- [ocp-schedules-18,1cf933f29adc154c,151d42a340a41fd1] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:27:50.649 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c44e85732fdad4eb,4a0f01ace8d09dda] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c44e85732fdad4eb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:50.649 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c44e85732fdad4eb,4a0f01ace8d09dda] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c44e85732fdad4eb, duration=2 ms] 2024-10-16 13:27:50.989 INFO 4486 --- [ocp-async-8,fd40ace826314499,58d9a9372852ee7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:51.062 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,098ae1a6bb31188d,df9d42191ca5e092] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=098ae1a6bb31188d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:51.062 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,098ae1a6bb31188d,df9d42191ca5e092] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=098ae1a6bb31188d, duration=1 ms] 2024-10-16 13:27:51.631 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:52.071 INFO 4486 --- [ocp-schedules-18,2f3d06aae0ef6721,a209c387b269d934] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:27:52.522 INFO 4486 --- [ocp-schedules-16,d5064ac37b49bffa,e13d7a552ccf51f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:27:52.522 INFO 4486 --- [ocp-schedules-16,48a777f88fa38562,5b8999b6b8b43837] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:27:52.524 INFO 4486 --- [ocp-schedules-16,48a777f88fa38562,5b8999b6b8b43837] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:27:52.555 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.555 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:52.565 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.566 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:52.592 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.592 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:52.600 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:52.601 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.601 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:27:52.635 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:52.636 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.637 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:52.638 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:27:52.638 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.642 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:27:52.643 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:52.652 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.652 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:52.663 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.673 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:52.674 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.674 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:52.675 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:27:52.680 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:27:52.685 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:52.687 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:52.694 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:52.695 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.696 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:27:52.706 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:27:52.726 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:27:52.726 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:27:52.747 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:52.749 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.750 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:27:52.753 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.756 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:27:52.759 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:52.780 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:27:52.786 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.786 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:52.789 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:52.790 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.791 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:27:52.795 INFO 4486 --- [ocp-schedules-25,e3c8443f2ad5f200,4f7f6492aebe369e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:27:52.801 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:27:52.801 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055886915000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:27:52.803 WARN 4486 --- [scanner-13235062-1,e3c8443f2ad5f200,d26a4f124c70ed16] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:27:52.816 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.816 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:52.844 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.844 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:52.877 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.877 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:27:52.908 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.908 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:27:52.939 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:27:52.965 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:52.966 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:52.967 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:27:52.971 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:27:52.975 INFO 4486 --- [ocp-async-10,b012983c8761605c,fb010f8dd3b9ffd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:27:52.976 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:27:52.981 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:27:52.983 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:27:52.990 INFO 4486 --- [ocp-async-7,27ad6a122f24af0b,992f0843d72993a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:52.994 INFO 4486 --- [prometheus-minute-schedule-1,52a7c28e67aad1f5,7072a382a0dfb89d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:52.999 INFO 4486 --- [prometheus-minute-schedule-1,52a7c28e67aad1f5,7072a382a0dfb89d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:53.009 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:27:53.152 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:27:53.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:27:53.181 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:27:53.182 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:27:53.209 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:27:53.209 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:27:54.631 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:54.990 INFO 4486 --- [ocp-async-1,91500cb299eec4d4,8bbdee1767e92ad4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:55.416 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,cd4a76a4b006da4b,1fd3b3d29b3055a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=cd4a76a4b006da4b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:27:55.417 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,cd4a76a4b006da4b,1fd3b3d29b3055a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=cd4a76a4b006da4b, duration=2 ms] 2024-10-16 13:27:55.994 INFO 4486 --- [prometheus-minute-schedule-1,1ee7cdb4b94f12bd,17583afb01f4f7b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:56.000 INFO 4486 --- [prometheus-minute-schedule-1,1ee7cdb4b94f12bd,17583afb01f4f7b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:27:56.696 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,74778fe4052a09c5,33d42e335c230008] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=74778fe4052a09c5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:27:56.696 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,74778fe4052a09c5,33d42e335c230008] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=74778fe4052a09c5, duration=1 ms] 2024-10-16 13:27:56.825 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4be42eaa42b98837,1bc9eb001c6693f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4be42eaa42b98837, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:56.825 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4be42eaa42b98837,1bc9eb001c6693f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4be42eaa42b98837, duration=1 ms] 2024-10-16 13:27:56.990 INFO 4486 --- [ocp-async-9,e37984c3800a68ca,8cb38d64f5d43df5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:57.632 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:27:58.410 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:27:58.427 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:27:58.429 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:27:58.453 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:27:58.455 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:27:58.479 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:27:58.481 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:27:58.509 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:27:58.511 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:27:58.539 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:27:58.541 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:27:58.571 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:27:58.573 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:27:58.603 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:27:58.605 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:27:58.631 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:27:58.633 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:27:58.663 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:27:58.666 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:27:58.696 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:27:58.698 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:27:58.722 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:27:58.723 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:27:58.744 INFO 4486 --- [ocp-schedules-6,644b42596b360e0b,0f8f28671bced1f3] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:27:58.775 INFO 4486 --- [ocp-schedules-8,5113502c6d63eec2,4d244211846403c9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:27:58.990 INFO 4486 --- [ocp-async-4,31ea869565509d71,6ab1d80297de5959] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:27:58.991 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,85ba76f2a907299d,1f87ad18c567d641] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=85ba76f2a907299d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:27:58.991 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,85ba76f2a907299d,1f87ad18c567d641] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=85ba76f2a907299d, duration=0 ms] 2024-10-16 13:27:58.994 INFO 4486 --- [prometheus-minute-schedule-1,bf0db90d783a4f87,658250638e27e740] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056480, interval=60 2024-10-16 13:27:59.000 INFO 4486 --- [prometheus-minute-schedule-1,bf0db90d783a4f87,658250638e27e740] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:00.106 INFO 4486 --- [ocp-schedules-25,bdb7aa617ffcb3e9,32277bb5f811e746] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:28:00.633 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:00.654 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c5e2e7ad960a1a5d,a58cb54583f163d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c5e2e7ad960a1a5d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:00.655 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c5e2e7ad960a1a5d,a58cb54583f163d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c5e2e7ad960a1a5d, duration=1 ms] 2024-10-16 13:28:00.750 INFO 4486 --- [ocp-schedules-22,f5842f3477b83b8b,5419c9c5049f7c0b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:28:00.750+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:28:00.755 INFO 4486 --- [ocp-schedules-22,f5842f3477b83b8b,5419c9c5049f7c0b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:28:00.768 INFO 4486 --- [ocp-schedules-22,f5842f3477b83b8b,5419c9c5049f7c0b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:28:00.780 INFO 4486 --- [ocp-schedules-22,f5842f3477b83b8b,5419c9c5049f7c0b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:28:00.990 INFO 4486 --- [ocp-async-3,a36b670878d6b2d9,fa006945ca4f0ffe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:01.067 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,90b82349ba0b5c55,56f884b0c5dcf750] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=90b82349ba0b5c55, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:01.067 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,90b82349ba0b5c55,56f884b0c5dcf750] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=90b82349ba0b5c55, duration=2 ms] 2024-10-16 13:28:01.994 INFO 4486 --- [prometheus-minute-schedule-1,beeab1c9e858d321,90bf9d079fed7590] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:02.001 INFO 4486 --- [prometheus-minute-schedule-1,beeab1c9e858d321,90bf9d079fed7590] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:02.923 INFO 4486 --- [ocp-schedules-13,f50488f0ec7f6be0,e3489b4eda70096d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:28:02.951 WARN 4486 --- [scanner-13235305-1,f50488f0ec7f6be0,b73fd0d91ae4c039] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:28:02.976 INFO 4486 --- [ocp-async-2,173b47c5cd818a46,986ba0c898cfc8e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:28:02.980 INFO 4486 --- [alarm-detect-1,f50488f0ec7f6be0,ee9ed9f0d9fcbf65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:02.986 INFO 4486 --- [alarm-detect-1,f50488f0ec7f6be0,ee9ed9f0d9fcbf65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:02.992 INFO 4486 --- [ocp-async-5,30bcad2f03db7b37,7dceea0294091abf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:02.993 INFO 4486 --- [alarm-detect-7,f50488f0ec7f6be0,9b8ffd23f573e825] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:02.993 INFO 4486 --- [alarm-detect-15,f50488f0ec7f6be0,7391e27af1385be1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:02.993 INFO 4486 --- [alarm-detect-1,f50488f0ec7f6be0,ee9ed9f0d9fcbf65] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:02.997 INFO 4486 --- [alarm-detect-7,f50488f0ec7f6be0,9b8ffd23f573e825] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:02.998 INFO 4486 --- [alarm-detect-15,f50488f0ec7f6be0,7391e27af1385be1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:03.001 INFO 4486 --- [alarm-detect-7,f50488f0ec7f6be0,9b8ffd23f573e825] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:03.003 INFO 4486 --- [alarm-detect-15,f50488f0ec7f6be0,7391e27af1385be1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:03.058 INFO 4486 --- [alarm-detect-1,f50488f0ec7f6be0,fa294e384b07d740] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056450 ~ 1729056472 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056465, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056465, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056465=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:28:03.111 INFO 4486 --- [alarm-detect-18,f50488f0ec7f6be0,32afb824bc494b3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:03.118 INFO 4486 --- [alarm-detect-18,f50488f0ec7f6be0,32afb824bc494b3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:03.125 INFO 4486 --- [alarm-detect-18,f50488f0ec7f6be0,32afb824bc494b3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:03.634 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:04.991 INFO 4486 --- [ocp-async-4,70ac456724d3ee9f,03cf05da32ec92ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:04.994 INFO 4486 --- [prometheus-minute-schedule-1,76bae23e5fb12be7,4239087f600bb934] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:05.002 INFO 4486 --- [prometheus-minute-schedule-1,76bae23e5fb12be7,4239087f600bb934] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:06.635 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:06.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,abdefc8adbbf79db,08247bc471234c89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=abdefc8adbbf79db, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:06.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,abdefc8adbbf79db,08247bc471234c89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=abdefc8adbbf79db, duration=2 ms] 2024-10-16 13:28:06.916 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,26e8baaa9d4e2183,215168f26d69fab5] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&version=2&ObClusterId=1701313637, client=10.197.49.184, traceId=26e8baaa9d4e2183, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=OceanBase,1701313637,2,{"ObRegion":"OceanBase","ObCluster":"OceanBase","ObRegionId":1701313637,"ObClusterId":1701313637,"Type":"PRIMARY","timestamp":1729056486911377,"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:28:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,26e8baaa9d4e2183,215168f26d69fab5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=OceanBase, obClusterId=1701313637, rootServerJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, groupName=OceanBase, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:28:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,26e8baaa9d4e2183,215168f26d69fab5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:28:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,26e8baaa9d4e2183,215168f26d69fab5] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=OceanBase, clusterId=1701313637, affectRows=1 2024-10-16 13:28:06.923 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,26e8baaa9d4e2183,215168f26d69fab5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.184, traceId=26e8baaa9d4e2183, duration=11 ms] 2024-10-16 13:28:06.991 INFO 4486 --- [ocp-async-7,efc44c67370baefb,155ec77eaeedfa63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:07.066 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:28:07.066 INFO 4486 --- [ocp-schedules-20,5af7b27c01480fc4,c73222bc9f5e46e6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:28:07.068 INFO 4486 --- [ocp-schedules-20,5af7b27c01480fc4,c73222bc9f5e46e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:07.081 INFO 4486 --- [ocp-schedules-12,a6090a4fc12da318,4e896a871cfd7853] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:28:07.083 INFO 4486 --- [ocp-schedules-12,a6090a4fc12da318,4e896a871cfd7853] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:28:07.090 INFO 4486 --- [ocp-schedules-6,836e464b1fdd7398,7c92b6d44eff94bd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:28:07.090 INFO 4486 --- [ocp-schedules-29,d6095f337a2268d4,a63f4ad1aaa45312] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:28:07.091 INFO 4486 --- [ocp-schedules-21,da73ee4d41e77a38,3390137ed896f3f5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:28:07.091 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:28:07.091 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:28:07.091 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:28:07.091 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:28:07.092 INFO 4486 --- [host-check-agent-status1,d6095f337a2268d4,7ee95429d59bd037] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:28:07.092 INFO 4486 --- [host-check-clockdiff1,da73ee4d41e77a38,743033c39b3f4d17] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:28:07.092 INFO 4486 --- [host-check-clockdiff2,da73ee4d41e77a38,29a09a165877cf50] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:28:07.092 INFO 4486 --- [host-check-clockdiff3,da73ee4d41e77a38,04f93aaffbeab7c1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:28:07.092 INFO 4486 --- [host-check-clockdiff4,da73ee4d41e77a38,07870aab640f3d71] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:28:07.092 INFO 4486 --- [host-check-agent-status2,d6095f337a2268d4,906c34ee5e7ad596] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:28:07.093 INFO 4486 --- [host-check-agent-status3,d6095f337a2268d4,288c3869ca2072b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:28:07.094 INFO 4486 --- [host-check-agent-status4,d6095f337a2268d4,c7564891ac1b1b6a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:28:07.098 INFO 4486 --- [host-check-agent-status1,d6095f337a2268d4,7ee95429d59bd037] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:07.099 INFO 4486 --- [host-check-agent-status2,d6095f337a2268d4,906c34ee5e7ad596] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:28:07.099 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:28:07.100 INFO 4486 --- [host-check-agent-status1,d6095f337a2268d4,7ee95429d59bd037] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:07.100 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:07.100 INFO 4486 --- [host-check-agent-status2,d6095f337a2268d4,906c34ee5e7ad596] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:28:07.122 INFO 4486 --- [host-check-agent-status4,d6095f337a2268d4,c7564891ac1b1b6a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:28:07.122 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:28:07.136 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:28:07.136 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:28:07.136 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:07.141 INFO 4486 --- [host-check-agent-status3,d6095f337a2268d4,288c3869ca2072b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:28:07.141 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:28:07.141 INFO 4486 --- [host-check-agent-status4,d6095f337a2268d4,c7564891ac1b1b6a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:28:07.141 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:28:07.141 INFO 4486 --- [host-check-clockdiff2,da73ee4d41e77a38,29a09a165877cf50] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056487 3 3 , error= 2024-10-16 13:28:07.141 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:07.141 INFO 4486 --- [host-check-clockdiff4,da73ee4d41e77a38,07870aab640f3d71] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056487 2 2 , error= 2024-10-16 13:28:07.141 INFO 4486 --- [host-check-clockdiff3,da73ee4d41e77a38,04f93aaffbeab7c1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056487 3 3 , error= 2024-10-16 13:28:07.145 INFO 4486 --- [host-check-clockdiff1,da73ee4d41e77a38,743033c39b3f4d17] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056487 0 0 , error= 2024-10-16 13:28:07.149 INFO 4486 --- [host-check-agent-status3,d6095f337a2268d4,288c3869ca2072b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:28:07.149 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:07.152 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:28:07.169 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:07.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:07.171 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.173 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:28:07.174 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:28:07.174 INFO 4486 --- [host-check-agent-status2,d6095f337a2268d4,906c34ee5e7ad596] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:28:07.174 INFO 4486 --- [host-check-agent-status3,d6095f337a2268d4,288c3869ca2072b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:28:07.174 INFO 4486 --- [host-check-agent-status4,d6095f337a2268d4,c7564891ac1b1b6a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:28:07.174 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:28:07.178 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:28:07.179 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:28:07.180 INFO 4486 --- [host-check-agent-status1,d6095f337a2268d4,7ee95429d59bd037] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:28:07.180 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:28:07.179633900+08:00 2024-10-16 13:28:07.180 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:28:07.180 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:28:07.180+08:00 2024-10-16 13:28:07.181 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:28:07.183659458+08:00 2024-10-16 13:28:07.181 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:28:07.183839938+08:00 2024-10-16 13:28:07.181 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:28:07.183851538+08:00 2024-10-16 13:28:07.182 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:28:07.181+08:00 2024-10-16 13:28:07.182 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:28:07.182+08:00 2024-10-16 13:28:07.182 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:28:07.181+08:00 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status3,d6095f337a2268d4,288c3869ca2072b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status3,d6095f337a2268d4,288c3869ca2072b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status3,d6095f337a2268d4,288c3869ca2072b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:28:07.180+08:00 and status AVAILABLE 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status4,d6095f337a2268d4,c7564891ac1b1b6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status4,d6095f337a2268d4,c7564891ac1b1b6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status4,d6095f337a2268d4,c7564891ac1b1b6a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status2,d6095f337a2268d4,906c34ee5e7ad596] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status2,d6095f337a2268d4,906c34ee5e7ad596] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:28:07.183 INFO 4486 --- [host-check-agent-status2,d6095f337a2268d4,906c34ee5e7ad596] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:28:07.184 INFO 4486 --- [host-check-agent-status1,d6095f337a2268d4,7ee95429d59bd037] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:28:07.184 INFO 4486 --- [host-check-agent-status1,d6095f337a2268d4,7ee95429d59bd037] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:28:07.184 INFO 4486 --- [host-check-agent-status1,d6095f337a2268d4,7ee95429d59bd037] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:28:07.184 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:28:07.181+08:00 and status AVAILABLE 2024-10-16 13:28:07.185 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:28:07.182+08:00 and status AVAILABLE 2024-10-16 13:28:07.185 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:28:07.181+08:00 and status AVAILABLE 2024-10-16 13:28:07.185 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:28:07.186 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:28:07.187 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:28:07.187 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:28:07.189 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.190 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:07.194 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:28:07.195 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:07.196 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:28:07.196 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:07.196 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:28:07.197 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:28:07.198 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:28:07.199 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:28:07.203 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.204 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:07.216 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.217 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:07.220 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.221 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:07.222 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:07.222 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:28:07.222 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:07.226 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:07.231 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:07.232 INFO 4486 --- [ocp-schedules-18,a026dd1f046e16b0,96b056abdc431394] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:28:07.250 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:28:07.260 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:28:07.261 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:28:07.265 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:28:07.271 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:28:07.278 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:07.279 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:28:07.285 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:28:07.288 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:07.296 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:28:07.297 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:07.351 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:28:07.351 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:28:07.351 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:28:07.362 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:28:07.362 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:28:07.363 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:07.364 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:28:07.381 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:07.381 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:07.386 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:07.411 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:28:07.417 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:28:07.418 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:28:07.421 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:28:07.422 INFO 4486 --- [host-check-host-status3,836e464b1fdd7398,f5733fae5a5b8675] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:28:07.428 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:28:07.428 INFO 4486 --- [host-check-host-status4,836e464b1fdd7398,2eb2cdaa81142af4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:28:07.431 INFO 4486 --- [host-check-host-status2,836e464b1fdd7398,219e657dcb7b20f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:28:07.444 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:07.494 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:28:07.505 INFO 4486 --- [host-check-host-status1,836e464b1fdd7398,0350e4ae68e5172a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:28:07.994 INFO 4486 --- [prometheus-minute-schedule-1,945d0d8893b08021,1a744671ff2500c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:08.001 INFO 4486 --- [prometheus-minute-schedule-1,945d0d8893b08021,1a744671ff2500c7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:08.991 INFO 4486 --- [ocp-async-2,65c3137d4e33e1de,2d8c9f2455a2e854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:08.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9a32431bcedf98aa,22cc1f3705c445ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9a32431bcedf98aa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:08.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9a32431bcedf98aa,22cc1f3705c445ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9a32431bcedf98aa, duration=1 ms] 2024-10-16 13:28:09.636 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:10.006 INFO 4486 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:28:10.006 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:28:10.029 INFO 4486 --- [Check obproxy2,3f865e4a73731a95,2c3452ab47115ab0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:28:10.029 INFO 4486 --- [Check obproxy2,3f865e4a73731a95,2c3452ab47115ab0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:10.029 INFO 4486 --- [Check obproxy1,ff660a894c9c56e0,3a3e448925c29d12] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:28:10.029 INFO 4486 --- [Check obproxy1,ff660a894c9c56e0,3a3e448925c29d12] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:10.034 INFO 4486 --- [Collect obproxy2,ffb069bf372a81b8,1ce3002782c4b8bb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:28:10.034 INFO 4486 --- [Collect obproxy1,04ebbbc50cbe3af8,619dff8159269b2a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:28:10.034 INFO 4486 --- [Collect obproxy2,ffb069bf372a81b8,1ce3002782c4b8bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:10.034 INFO 4486 --- [Collect obproxy1,04ebbbc50cbe3af8,619dff8159269b2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:10.051 INFO 4486 --- [Check obproxy2,3f865e4a73731a95,2c3452ab47115ab0] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:28:10.051 INFO 4486 --- [Check obproxy2,3f865e4a73731a95,2c3452ab47115ab0] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:28:10.054 INFO 4486 --- [Collect obproxy1,04ebbbc50cbe3af8,619dff8159269b2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:28:10.055 INFO 4486 --- [Collect obproxy2,ffb069bf372a81b8,1ce3002782c4b8bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:28:10.059 INFO 4486 --- [Collect obproxy1,04ebbbc50cbe3af8,619dff8159269b2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:28:10.060 INFO 4486 --- [Collect obproxy2,ffb069bf372a81b8,1ce3002782c4b8bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:28:10.061 INFO 4486 --- [Check obproxy1,ff660a894c9c56e0,3a3e448925c29d12] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:28:10.061 INFO 4486 --- [Check obproxy1,ff660a894c9c56e0,3a3e448925c29d12] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:28:10.061 INFO 4486 --- [Collect obproxy1,04ebbbc50cbe3af8,619dff8159269b2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:28:10.064 INFO 4486 --- [Collect obproxy2,ffb069bf372a81b8,1ce3002782c4b8bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:28:10.111 INFO 4486 --- [ocp-schedules-3,974f058fd5833c0e,f03a19070a0afc2e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:28:10.659 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e9159c90e0ba72eb,2a4833a907cf952d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e9159c90e0ba72eb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:10.659 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e9159c90e0ba72eb,2a4833a907cf952d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e9159c90e0ba72eb, duration=1 ms] 2024-10-16 13:28:10.977 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:28:10.992 INFO 4486 --- [ocp-async-10,462df1ed06d7131d,1766feb8328a8424] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:10.994 INFO 4486 --- [prometheus-minute-schedule-1,3b927e8e038e2928,8435752538d0ef2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:11.001 INFO 4486 --- [prometheus-minute-schedule-1,3b927e8e038e2928,8435752538d0ef2f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:28:11.071 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9bf7f5736745e59e,88d5b166cdc7bd17] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9bf7f5736745e59e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:11.071 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9bf7f5736745e59e,88d5b166cdc7bd17] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9bf7f5736745e59e, duration=1 ms] 2024-10-16 13:28:12.637 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:12.907 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056420, nodeCount=449, affectRows=449 2024-10-16 13:28:12.910 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056420, nodeCount=448, validCount=448 2024-10-16 13:28:12.931 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056420, nodeCount=1000, affectRows=1000 2024-10-16 13:28:12.932 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056420, nodeCount=1000, affectRows=1000 2024-10-16 13:28:12.933 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056420, nodeCount=1000, affectRows=1000 2024-10-16 13:28:12.936 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056420, nodeCount=1000, validCount=1000 2024-10-16 13:28:12.937 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056420, nodeCount=998, validCount=998 2024-10-16 13:28:12.937 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056420, nodeCount=1000, validCount=1000 2024-10-16 13:28:12.976 INFO 4486 --- [ocp-async-6,18df261da603b695,a5de25e2131f8531] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:28:12.992 INFO 4486 --- [ocp-async-3,8a69aed069e10ecf,079ea917bec5eda3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:13.154 INFO 4486 --- [ocp-schedules-6,f81de6a9006dfc2d,d16ba52a92be5f5c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:28:13.196 WARN 4486 --- [scanner-13236469-1,f81de6a9006dfc2d,b9ef15ee4156c6fa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:28:13.245 INFO 4486 --- [alarm-detect-14,f81de6a9006dfc2d,8bff96a31de515b0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056465 ~ 1729056483 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056480, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056480, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056480=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:28:13.994 INFO 4486 --- [prometheus-minute-schedule-1,150e7de82f3824a7,aa4fb02ead4bd909] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:14.001 INFO 4486 --- [prometheus-minute-schedule-1,150e7de82f3824a7,aa4fb02ead4bd909] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:14.280 INFO 4486 --- [metric-archive-async-26,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056420, nodeCount=223, affectRows=223 2024-10-16 13:28:14.281 INFO 4486 --- [metric-archive-async-18,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056420, nodeCount=254, affectRows=254 2024-10-16 13:28:14.586 INFO 4486 --- [metric-archive-async-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056420, nodeCount=223, validCount=223 2024-10-16 13:28:14.622 INFO 4486 --- [metric-archive-async-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056420, nodeCount=254, validCount=254 2024-10-16 13:28:14.992 INFO 4486 --- [ocp-async-9,79f5c86db6bb3abb,2ad349233485411c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:15.637 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:16.837 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3e3caaadc2d28c59,06c190a1a5b50f8d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3e3caaadc2d28c59, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:16.837 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3e3caaadc2d28c59,06c190a1a5b50f8d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3e3caaadc2d28c59, duration=1 ms] 2024-10-16 13:28:16.992 INFO 4486 --- [ocp-async-5,1c97bb4167815e72,bb79a8f26dfb14c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:16.994 INFO 4486 --- [prometheus-minute-schedule-1,87bfeadeeed1dc22,261adc3ad47b2e59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:17.000 INFO 4486 --- [prometheus-minute-schedule-1,87bfeadeeed1dc22,261adc3ad47b2e59] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:17.049 INFO 4486 --- [ocp-schedules-4,0429d6a106223562,57cdae1b8d021963] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:28:17.070 INFO 4486 --- [ocp-schedules-20,5af7b27c01480fc4,c73222bc9f5e46e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:28:17.179 INFO 4486 --- [ocp-schedules-12,a6090a4fc12da318,4e896a871cfd7853] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:28:18.638 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:18.780 INFO 4486 --- [ocp-schedules-18,de4296899ff605e0,1ad3cc1caa313850] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:28:18.992 INFO 4486 --- [ocp-async-6,4c58909c3e767489,2cb85a8e3990aac2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:19.001 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,56ab7c9d2ad9f9d5,d9023954a75894c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=56ab7c9d2ad9f9d5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:19.001 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,56ab7c9d2ad9f9d5,d9023954a75894c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=56ab7c9d2ad9f9d5, duration=1 ms] 2024-10-16 13:28:19.994 INFO 4486 --- [prometheus-minute-schedule-1,865c38534663bd12,85989a278243f497] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:20.000 INFO 4486 --- [prometheus-minute-schedule-1,865c38534663bd12,85989a278243f497] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:20.116 INFO 4486 --- [ocp-schedules-9,420943e89249311a,f7d7c7f0ae627404] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:28:20.669 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9e4c7862a8458694,5ec65d4e5d242f01] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9e4c7862a8458694, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:20.669 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9e4c7862a8458694,5ec65d4e5d242f01] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9e4c7862a8458694, duration=2 ms] 2024-10-16 13:28:20.810 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:28:20.993 INFO 4486 --- [ocp-async-1,eacb2bd162532d33,9c8719b4a7198cc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:20.996 INFO 4486 --- [prometheus-cache-monitor1,f33e4edf97704146,0b1602a45e47807d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13854,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:28:20.997 INFO 4486 --- [prometheus-cache-monitor1,f33e4edf97704146,0b1602a45e47807d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:28:21.077 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,16a93f122baabe29,1e78dab7245b5a90] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=16a93f122baabe29, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:21.077 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,16a93f122baabe29,1e78dab7245b5a90] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=16a93f122baabe29, duration=1 ms] 2024-10-16 13:28:21.639 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:22.543 INFO 4486 --- [ocp-schedules-7,d8e314ce0ee74492,f398ca7edc33172e] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:28:22.976 INFO 4486 --- [ocp-async-9,7c8cfd415c09fc27,2ec021c52259c86d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:28:22.993 INFO 4486 --- [ocp-async-8,abd27a5252a9036f,08fd73ed553b9e46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:22.994 INFO 4486 --- [prometheus-minute-schedule-1,0938004af041d890,524ec694b948648b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:23.000 INFO 4486 --- [prometheus-minute-schedule-1,0938004af041d890,524ec694b948648b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:23.026 INFO 4486 --- [ocp-schedules-12,9ac018f900a715f1,5fe6c37fef777c19] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:28:23.027 INFO 4486 --- [ocp-schedules-12,9ac018f900a715f1,5fe6c37fef777c19] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:23.064 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:28:23.065 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.069 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:28:23.077 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.078 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:28:23.079 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:28:23.080 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.085 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:28:23.085 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.086 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:28:23.094 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.095 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:28:23.100 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.101 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:28:23.103 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.103 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:28:23.111 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.112 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.112 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:23.112 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:28:23.117 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:28:23.118 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.119 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.120 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:23.122 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:28:23.124 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.125 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:23.127 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.128 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:28:23.133 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.134 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:28:23.135 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.136 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.136 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:23.136 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:28:23.146 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.147 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.147 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:28:23.148 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:28:23.155 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.156 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.156 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.157 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.158 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:28:23.158 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.158 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.159 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.160 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.161 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.161 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.161 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.162 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:28:23.162 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.162 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.163 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.163 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:28:23.164 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.164 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.165 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.166 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.167 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.167 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.168 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.169 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.170 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.171 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.171 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.171 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:28:23.172 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.172 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.173 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.174 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:28:23.177 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:28:23.181 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.182 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:28:23.184 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.185 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:23.187 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.189 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.191 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.194 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.196 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.196 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.196 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.196 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.197 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:23.197 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:28:23.198 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.200 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.201 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.204 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.205 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.205 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.206 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:28:23.206 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.206 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:28:23.206 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.206 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:28:23.206 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.206 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:28:23.207 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.208 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:23.209 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.210 INFO 4486 --- [ocp-schedules-16,48a777f88fa38562,5b8999b6b8b43837] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:28:23.210 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.210 INFO 4486 --- [ocp-schedules-16,a27434908d9f751a,4645ff58b1962fe8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:28:23.210 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:28:23.210 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:28:23.211 INFO 4486 --- [ocp-schedules-16,a27434908d9f751a,4645ff58b1962fe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:23.212 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.213 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.217 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.219 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.221 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.222 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.224 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:28:23.226 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:28:23.236 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.237 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:23.238 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.239 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.242 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.243 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.246 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.247 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.248 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.248 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:23.251 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.251 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.252 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.253 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.254 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:28:23.255 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.256 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.258 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.259 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.262 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.263 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:28:23.264 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.264 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:28:23.264 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.264 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:28:23.264 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.264 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:28:23.266 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.268 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:28:23.268 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:28:23.268 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:28:23.268 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.269 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.269 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.269 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:23.270 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:28:23.270 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.271 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:28:23.273 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:28:23.276 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:28:23.300 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.301 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.302 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:23.303 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:28:23.303 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.343 INFO 4486 --- [ocp-schedules-29,f47d56be3f0f731d,0c625a0bee58ccef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:28:23.345 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:28:23.346 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:23.359 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.359 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:23.374 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.379 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.381 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.382 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:28:23.390 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.391 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.391 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:23.392 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.392 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:28:23.392 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.392 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:28:23.392 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:28:23.392 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:28:23.392 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:28:23.395 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:28:23.395 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:28:23.400 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:28:23.408 WARN 4486 --- [scanner-13236723-1,f47d56be3f0f731d,8fc8b0bd78e926bf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:28:23.413 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:23.420 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:23.440 INFO 4486 --- [alarm-detect-7,f47d56be3f0f731d,1a57c5026c6a0fd5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:23.446 INFO 4486 --- [alarm-detect-7,f47d56be3f0f731d,1a57c5026c6a0fd5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:23.451 INFO 4486 --- [alarm-detect-7,f47d56be3f0f731d,1a57c5026c6a0fd5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:23.463 INFO 4486 --- [alarm-detect-10,f47d56be3f0f731d,5db16810544f38d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:23.463 INFO 4486 --- [alarm-detect-12,f47d56be3f0f731d,65633de93fe46db0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:23.467 INFO 4486 --- [alarm-detect-10,f47d56be3f0f731d,5db16810544f38d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:23.468 INFO 4486 --- [alarm-detect-12,f47d56be3f0f731d,65633de93fe46db0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:23.472 INFO 4486 --- [alarm-detect-10,f47d56be3f0f731d,5db16810544f38d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:23.473 INFO 4486 --- [alarm-detect-12,f47d56be3f0f731d,65633de93fe46db0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:23.479 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:28:23.481 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.483 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.484 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:28:23.568 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:28:23.572 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:28:23.577 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:28:23.591 INFO 4486 --- [alarm-detect-2,f47d56be3f0f731d,4aaae376f4026776] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:23.596 INFO 4486 --- [alarm-detect-2,f47d56be3f0f731d,4aaae376f4026776] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:23.601 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.601 INFO 4486 --- [alarm-detect-2,f47d56be3f0f731d,4aaae376f4026776] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:23.601 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.602 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.603 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:28:23.603 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.604 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:23.608 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.612 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:28:23.614 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:28:23.615 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:28:23.616 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:23.645 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.645 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:23.675 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.675 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:23.700 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.700 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:23.729 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.729 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:23.756 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.756 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:23.785 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:23.813 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:23.815 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:23.816 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:23.820 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:28:23.825 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:28:23.830 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:28:23.832 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:28:23.853 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:28:23.997 INFO 4486 --- [ocp-schedules-13,d2111b4d5cee0613,03a4e240751ae34e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:28:23.998 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:28:24.027 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:24.028 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:24.029 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:28:24.057 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:28:24.057 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:28:24.085 INFO 4486 --- [ocp-schedules-30,8bf0dc1fd356d534,80b144f3a31ba85a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:28:24.085 INFO 4486 --- [ocp-schedules-30,8bf0dc1fd356d534,80b144f3a31ba85a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:28:24.117 INFO 4486 --- [Collect all tenants major compactions1,8bf0dc1fd356d534,5161da4ed823002f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:28:24.117 INFO 4486 --- [Collect all tenants major compactions2,8bf0dc1fd356d534,f7a0632222367059] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:28:24.117 INFO 4486 --- [Collect all tenants major compactions4,8bf0dc1fd356d534,bc33ae0cd92a69ca] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:28:24.117 INFO 4486 --- [Collect all tenants major compactions3,8bf0dc1fd356d534,7b140ed82d172d78] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:28:24.118 INFO 4486 --- [Collect all tenants major compactions5,8bf0dc1fd356d534,c5680ee2a1d9c1c7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:28:24.129 INFO 4486 --- [Collect all tenants major compactions1,8bf0dc1fd356d534,5161da4ed823002f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:24.129 INFO 4486 --- [Collect all tenants major compactions2,8bf0dc1fd356d534,f7a0632222367059] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:24.130 INFO 4486 --- [Collect all tenants major compactions1,8bf0dc1fd356d534,5161da4ed823002f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:28:24.131 INFO 4486 --- [Collect all tenants major compactions2,8bf0dc1fd356d534,f7a0632222367059] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:28:24.133 INFO 4486 --- [Collect all tenants major compactions1,8bf0dc1fd356d534,5161da4ed823002f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:28:24.133 INFO 4486 --- [Collect all tenants major compactions2,8bf0dc1fd356d534,f7a0632222367059] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:28:24.133 INFO 4486 --- [Collect all tenants major compactions1,8bf0dc1fd356d534,5161da4ed823002f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:28:24.133 INFO 4486 --- [Collect all tenants major compactions2,8bf0dc1fd356d534,f7a0632222367059] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:28:24.134 INFO 4486 --- [Collect all tenants major compactions3,8bf0dc1fd356d534,7b140ed82d172d78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:24.136 INFO 4486 --- [Collect all tenants major compactions3,8bf0dc1fd356d534,7b140ed82d172d78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:28:24.139 INFO 4486 --- [Collect all tenants major compactions3,8bf0dc1fd356d534,7b140ed82d172d78] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:28:24.139 INFO 4486 --- [Collect all tenants major compactions3,8bf0dc1fd356d534,7b140ed82d172d78] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:28:24.148 INFO 4486 --- [Collect all tenants major compactions5,8bf0dc1fd356d534,c5680ee2a1d9c1c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:24.149 INFO 4486 --- [Collect all tenants major compactions4,8bf0dc1fd356d534,bc33ae0cd92a69ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:24.150 INFO 4486 --- [Collect all tenants major compactions4,8bf0dc1fd356d534,bc33ae0cd92a69ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:28:24.151 INFO 4486 --- [Collect all tenants major compactions5,8bf0dc1fd356d534,c5680ee2a1d9c1c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:28:24.153 INFO 4486 --- [Collect all tenants major compactions4,8bf0dc1fd356d534,bc33ae0cd92a69ca] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:28:24.154 INFO 4486 --- [Collect all tenants major compactions4,8bf0dc1fd356d534,bc33ae0cd92a69ca] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:28:24.156 INFO 4486 --- [Collect all tenants major compactions5,8bf0dc1fd356d534,c5680ee2a1d9c1c7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:28:24.156 INFO 4486 --- [Collect all tenants major compactions5,8bf0dc1fd356d534,c5680ee2a1d9c1c7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:28:24.157 INFO 4486 --- [ocp-schedules-30,8bf0dc1fd356d534,80b144f3a31ba85a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:28:24.640 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:24.993 INFO 4486 --- [ocp-async-4,ebcf616a1c3f15e7,eadcd6810a201144] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:25.994 INFO 4486 --- [prometheus-minute-schedule-1,92f608eb2e541153,cf8408b396711307] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:25.999 INFO 4486 --- [prometheus-minute-schedule-1,92f608eb2e541153,cf8408b396711307] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:26.672 INFO 4486 --- [ocp-schedules-22,c93b1eb48a522c8f,dfa67d203df5f268] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:28:26.672+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:28:26.674 INFO 4486 --- [ocp-schedules-22,c93b1eb48a522c8f,dfa67d203df5f268] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:28:26.675 INFO 4486 --- [ocp-schedules-22,c93b1eb48a522c8f,dfa67d203df5f268] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:28:26.675 INFO 4486 --- [ocp-schedules-22,c93b1eb48a522c8f,dfa67d203df5f268] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:28:26.676 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ea63ba5c0cfb75d5,d7cfab9d1b6ff769] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=ea63ba5c0cfb75d5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:28:26.677 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ea63ba5c0cfb75d5,d7cfab9d1b6ff769] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=ea63ba5c0cfb75d5, duration=2 ms] 2024-10-16 13:28:26.841 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,37dcb7d47a3a4e09,3a7abf26b690136a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=37dcb7d47a3a4e09, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:26.842 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,37dcb7d47a3a4e09,3a7abf26b690136a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=37dcb7d47a3a4e09, duration=1 ms] 2024-10-16 13:28:26.993 INFO 4486 --- [ocp-async-2,75f2884ddbb966cb,70241c01e2943b07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:27.641 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:28.993 INFO 4486 --- [ocp-async-10,a9f62a0c258b8fc1,0810ce886dce910b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:28.994 INFO 4486 --- [prometheus-minute-schedule-1,3f19cf422f30dc66,5e981c27475f2cec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:29.000 INFO 4486 --- [prometheus-minute-schedule-1,3f19cf422f30dc66,5e981c27475f2cec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:29.006 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,64c73895e3d9bb8d,d58b4f41460411b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=64c73895e3d9bb8d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:29.006 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,64c73895e3d9bb8d,d58b4f41460411b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=64c73895e3d9bb8d, duration=1 ms] 2024-10-16 13:28:30.103 INFO 4486 --- [prometheus-second-schedule-1,6481e7139e2192ea,b1d6cc845364e479] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:28:30.122 INFO 4486 --- [ocp-schedules-20,0a5108bd336e32b1,2d64b505c7cfde73] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:28:30.642 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:30.675 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f46aa81b1559fa50,758db74e5cd932a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f46aa81b1559fa50, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:30.675 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f46aa81b1559fa50,758db74e5cd932a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f46aa81b1559fa50, duration=1 ms] 2024-10-16 13:28:30.994 INFO 4486 --- [ocp-async-7,b3e27ac28f7540cf,22f7cb45adedc4f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:31.081 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,af39ca5b04e0db61,cda48abee37bd962] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=af39ca5b04e0db61, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:31.081 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,af39ca5b04e0db61,cda48abee37bd962] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=af39ca5b04e0db61, duration=1 ms] 2024-10-16 13:28:31.994 INFO 4486 --- [prometheus-minute-schedule-1,8bca63a158eea2cb,59e475eae9cec497] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:32.000 INFO 4486 --- [prometheus-minute-schedule-1,8bca63a158eea2cb,59e475eae9cec497] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:32.976 INFO 4486 --- [ocp-async-6,b8cff4a26628659f,4e520d174eb5e910] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:28:32.994 INFO 4486 --- [ocp-async-3,9a1204520731a675,5e3a563e4167ac00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:33.437 INFO 4486 --- [ocp-schedules-20,710692fccecc9eb8,c7ea1986b1fb2c9a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:28:33.439 INFO 4486 --- [ocp-schedules-20,710692fccecc9eb8,c7ea1986b1fb2c9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:33.440 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:28:33.440 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:28:33.440 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:28:33.625 INFO 4486 --- [ocp-schedules-23,597a64ae7150ed0b,a7db26b9887f4277] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:28:33.643 WARN 4486 --- [scanner-13237033-1,597a64ae7150ed0b,7ff62ccb829b1adc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:28:33.643 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:33.711 INFO 4486 --- [alarm-detect-6,597a64ae7150ed0b,6621e6bcc128ef9e] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056480 ~ 1729056503 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056495, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056495, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056495=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:28:33.715 INFO 4486 --- [alarm-detect-6,597a64ae7150ed0b,6621e6bcc128ef9e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:28:34.994 INFO 4486 --- [ocp-async-9,557d65129a8893ab,b7068de66d45f7f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:34.994 INFO 4486 --- [prometheus-minute-schedule-1,c46b19ee9a97c4e9,9c90f48e30d71825] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:35.004 INFO 4486 --- [prometheus-minute-schedule-1,c46b19ee9a97c4e9,9c90f48e30d71825] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:36.644 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:36.847 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c1ae7499c71057f8,5877907651735299] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c1ae7499c71057f8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:36.847 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c1ae7499c71057f8,5877907651735299] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c1ae7499c71057f8, duration=2 ms] 2024-10-16 13:28:36.994 INFO 4486 --- [ocp-async-5,56d3c7afc226f50f,54962dc660449bd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:37.063 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:28:37.064 INFO 4486 --- [ocp-schedules-8,2159dc039fcc4708,874a63c493d3aea1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:28:37.066 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:28:37.067 INFO 4486 --- [ocp-schedules-11,960c79d7ea11e005,c2a16b613bbb7a06] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:28:37.068 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:28:37.068 INFO 4486 --- [ocp-schedules-11,960c79d7ea11e005,c2a16b613bbb7a06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:37.070 INFO 4486 --- [ocp-schedules-8,2159dc039fcc4708,874a63c493d3aea1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:28:37.081 INFO 4486 --- [ocp-schedules-27,37f1de15bf77fe0f,4a6369d4bd3bdcb2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:28:37.083 INFO 4486 --- [ocp-schedules-27,37f1de15bf77fe0f,4a6369d4bd3bdcb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:28:37.089 INFO 4486 --- [ocp-schedules-21,011b1692ca1d4364,93d877531b84e259] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:28:37.089 INFO 4486 --- [ocp-schedules-18,0061920b990f84d2,05e151865dc0fb9d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:28:37.090 INFO 4486 --- [ocp-schedules-6,73dd4aaaacbc50c2,96795ae09dac4839] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:28:37.090 INFO 4486 --- [host-check-clockdiff1,0061920b990f84d2,e61f0e9c10693e39] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:28:37.090 INFO 4486 --- [host-check-agent-status1,011b1692ca1d4364,3985880a0d45872b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:28:37.091 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:28:37.092 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:28:37.092 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:28:37.092 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:28:37.092 INFO 4486 --- [host-check-clockdiff2,0061920b990f84d2,40f57dc09fdd0f08] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:28:37.093 INFO 4486 --- [host-check-clockdiff3,0061920b990f84d2,826dc9d5dc019a50] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:28:37.093 INFO 4486 --- [host-check-agent-status4,011b1692ca1d4364,0da3f2939e01a3ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:28:37.094 INFO 4486 --- [host-check-clockdiff4,0061920b990f84d2,a654e8602a3cf2a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:28:37.094 INFO 4486 --- [host-check-agent-status3,011b1692ca1d4364,9d6e3c7c9bce6fa8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:28:37.094 INFO 4486 --- [host-check-agent-status2,011b1692ca1d4364,6d929dba2e50c9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:28:37.100 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:28:37.100 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:37.100 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:28:37.101 INFO 4486 --- [host-check-agent-status2,011b1692ca1d4364,6d929dba2e50c9e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:28:37.101 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:28:37.101 INFO 4486 --- [host-check-agent-status4,011b1692ca1d4364,0da3f2939e01a3ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:28:37.101 INFO 4486 --- [host-check-agent-status1,011b1692ca1d4364,3985880a0d45872b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:37.103 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:37.103 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:28:37.103 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:28:37.103 INFO 4486 --- [host-check-agent-status2,011b1692ca1d4364,6d929dba2e50c9e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:28:37.104 INFO 4486 --- [host-check-agent-status4,011b1692ca1d4364,0da3f2939e01a3ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:28:37.103 INFO 4486 --- [host-check-agent-status1,011b1692ca1d4364,3985880a0d45872b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:37.103 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:37.103 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:28:37.105 INFO 4486 --- [host-check-agent-status3,011b1692ca1d4364,9d6e3c7c9bce6fa8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:28:37.153 INFO 4486 --- [host-check-clockdiff1,0061920b990f84d2,e61f0e9c10693e39] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056517 0 0 , error= 2024-10-16 13:28:37.164 INFO 4486 --- [host-check-clockdiff3,0061920b990f84d2,826dc9d5dc019a50] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056517 3 3 , error= 2024-10-16 13:28:37.165 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.168 INFO 4486 --- [host-check-clockdiff4,0061920b990f84d2,a654e8602a3cf2a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056517 3 3 , error= 2024-10-16 13:28:37.168 INFO 4486 --- [host-check-agent-status3,011b1692ca1d4364,9d6e3c7c9bce6fa8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:28:37.168 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:37.169 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:28:37.190 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.191 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:37.192 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:37.193 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:28:37.194 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.194 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:28:37.194 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:28:37.195 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:28:37.196 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.196 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:28:37.196 INFO 4486 --- [host-check-agent-status1,011b1692ca1d4364,3985880a0d45872b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:28:37.196 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.196 INFO 4486 --- [host-check-agent-status4,011b1692ca1d4364,0da3f2939e01a3ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:28:37.201 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:28:37.199416052+08:00 2024-10-16 13:28:37.201 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:28:37.199204926+08:00 2024-10-16 13:28:37.201 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:28:37.201 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:28:37.201+08:00 2024-10-16 13:28:37.202 INFO 4486 --- [host-check-agent-status3,011b1692ca1d4364,9d6e3c7c9bce6fa8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:28:37.202 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:28:37.201+08:00 2024-10-16 13:28:37.202 INFO 4486 --- [host-check-agent-status2,011b1692ca1d4364,6d929dba2e50c9e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:28:37.202 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:28:37.204893296+08:00 2024-10-16 13:28:37.202 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:28:37.202+08:00 2024-10-16 13:28:37.203 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:28:37.203 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.204 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:28:37.201+08:00 and status AVAILABLE 2024-10-16 13:28:37.204 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:28:37.201+08:00 and status AVAILABLE 2024-10-16 13:28:37.205 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:28:37.202+08:00 and status AVAILABLE 2024-10-16 13:28:37.205 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:28:37.205600471+08:00 2024-10-16 13:28:37.205 INFO 4486 --- [host-check-agent-status1,011b1692ca1d4364,3985880a0d45872b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:28:37.205 INFO 4486 --- [host-check-agent-status1,011b1692ca1d4364,3985880a0d45872b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:28:37.205 INFO 4486 --- [host-check-agent-status1,011b1692ca1d4364,3985880a0d45872b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:28:37.205 INFO 4486 --- [host-check-agent-status3,011b1692ca1d4364,9d6e3c7c9bce6fa8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:28:37.205 INFO 4486 --- [host-check-agent-status3,011b1692ca1d4364,9d6e3c7c9bce6fa8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-agent-status3,011b1692ca1d4364,9d6e3c7c9bce6fa8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-agent-status4,011b1692ca1d4364,0da3f2939e01a3ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-agent-status4,011b1692ca1d4364,0da3f2939e01a3ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-agent-status4,011b1692ca1d4364,0da3f2939e01a3ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:28:37.206+08:00 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-agent-status2,011b1692ca1d4364,6d929dba2e50c9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-agent-status2,011b1692ca1d4364,6d929dba2e50c9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:28:37.206 INFO 4486 --- [host-check-agent-status2,011b1692ca1d4364,6d929dba2e50c9e3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:28:37.207 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:28:37.207 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:28:37.207 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:28:37.208 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:28:37.209 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:28:37.206+08:00 and status AVAILABLE 2024-10-16 13:28:37.211 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:28:37.216 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:28:37.216 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:28:37.217 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.218 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:28:37.218 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:28:37.218 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:28:37.218 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:37.220 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:37.220 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:28:37.221 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:37.224 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.224 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.224 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.224 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.224 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.224 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.228 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.229 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:37.241 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.242 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:37.242 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:28:37.243 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:37.244 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:28:37.244 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:37.244 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.245 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:28:37.248 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:28:37.255 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.256 INFO 4486 --- [ocp-schedules-1,b4034f282b4050d7,072ba979b95a1c76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:28:37.279 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:28:37.280 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:28:37.290 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:28:37.290 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:28:37.290 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:28:37.296 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:28:37.300 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:28:37.306 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:28:37.307 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:37.307 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:37.317 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:37.322 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:28:37.337 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:28:37.347 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:28:37.351 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:28:37.355 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:28:37.362 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:28:37.365 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:37.367 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:28:37.380 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:37.381 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:28:37.383 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:37.393 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:28:37.395 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:28:37.405 INFO 4486 --- [host-check-host-status3,73dd4aaaacbc50c2,2b52a88bf648f63d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:28:37.410 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:28:37.416 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:28:37.418 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:28:37.427 INFO 4486 --- [host-check-host-status2,73dd4aaaacbc50c2,0ecaa7207a47c7ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:28:37.430 INFO 4486 --- [host-check-host-status4,73dd4aaaacbc50c2,6ab6375a320ee8d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:28:37.465 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:28:37.475 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:37.477 INFO 4486 --- [host-check-host-status1,73dd4aaaacbc50c2,df287978817ebd00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:28:37.477 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.481 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:28:37.483 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.484 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.485 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:28:37.499 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.499 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.499 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.499 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.499 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.501 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.529 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:28:37.594 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:28:37.642 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:37.659 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:37.660 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.661 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:28:37.662 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:28:37.665 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.666 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:28:37.667 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.668 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.669 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:28:37.678 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.678 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.678 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.678 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.678 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.678 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.686 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:28:37.923 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:28:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:28:37.977 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:37.979 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.980 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:28:37.981 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.982 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.983 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:28:37.992 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.992 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.992 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.992 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.992 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:28:37.993 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:37.994 INFO 4486 --- [prometheus-minute-schedule-1,2d7208611919bd13,81af663a8a4e4b99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:38.001 INFO 4486 --- [prometheus-minute-schedule-1,2d7208611919bd13,81af663a8a4e4b99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:38.012 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:28:38.083 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9204b63818bfdba0,367430a0d4ede0bd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=9204b63818bfdba0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:28:38.084 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9204b63818bfdba0,367430a0d4ede0bd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=9204b63818bfdba0, duration=2 ms] 2024-10-16 13:28:38.215 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:28:38.257 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:38.259 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:38.259 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:28:38.260 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:38.261 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:38.262 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:28:38.272 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:38.272 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:38.272 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:38.272 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:38.272 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:28:38.272 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:38.336 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:28:38.542 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:28:38.340084, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:28:38.542 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:28:38.340084, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:28:38.582 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:28:38.340084} } 2024-10-16 13:28:38.634 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:28:38.641 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:28:38.644 INFO 4486 --- [ocp-schedules-22,96f81668f82f81f8,d77a5fce305dda27] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:28:38.786 INFO 4486 --- [ocp-schedules-10,ed90fab12d10bcb6,2febcdd0ed679d0f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:28:38.901 INFO 4486 --- [host-check-clockdiff2,0061920b990f84d2,40f57dc09fdd0f08] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056518 3 3 , error= 2024-10-16 13:28:38.995 INFO 4486 --- [ocp-async-1,cfbed1b3ad4d4470,fbfa46f7d7120733] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:39.058 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e6b05e6c9f4a21b9,287e6c8ad4c2b9d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e6b05e6c9f4a21b9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:39.059 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e6b05e6c9f4a21b9,287e6c8ad4c2b9d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e6b05e6c9f4a21b9, duration=1 ms] 2024-10-16 13:28:39.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6e84925b876bd2ed,d1a675e7b3ad0279] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=6e84925b876bd2ed, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:28:39.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6e84925b876bd2ed,d1a675e7b3ad0279] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=6e84925b876bd2ed, duration=1 ms] 2024-10-16 13:28:39.645 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:40.007 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:28:40.007 INFO 4486 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:28:40.025 INFO 4486 --- [Check obproxy1,a8606a90e92f2f5f,a1d524d9d7401265] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:28:40.025 INFO 4486 --- [Check obproxy2,e0e9a905f2e2a613,b8c29634ef3f702e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:28:40.026 INFO 4486 --- [Check obproxy1,a8606a90e92f2f5f,a1d524d9d7401265] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:40.026 INFO 4486 --- [Check obproxy2,e0e9a905f2e2a613,b8c29634ef3f702e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:40.031 INFO 4486 --- [Collect obproxy2,fbb56d58860f2e46,63388d3b543444c4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:28:40.031 INFO 4486 --- [Collect obproxy2,fbb56d58860f2e46,63388d3b543444c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:40.032 INFO 4486 --- [Collect obproxy1,081357183666fae9,22a62a581ee7f7bb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:28:40.032 INFO 4486 --- [Collect obproxy1,081357183666fae9,22a62a581ee7f7bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:28:40.047 INFO 4486 --- [Check obproxy2,e0e9a905f2e2a613,b8c29634ef3f702e] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:28:40.047 INFO 4486 --- [Check obproxy2,e0e9a905f2e2a613,b8c29634ef3f702e] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:28:40.054 INFO 4486 --- [Collect obproxy2,fbb56d58860f2e46,63388d3b543444c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:28:40.059 INFO 4486 --- [Collect obproxy2,fbb56d58860f2e46,63388d3b543444c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:28:40.063 INFO 4486 --- [Check obproxy1,a8606a90e92f2f5f,a1d524d9d7401265] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:28:40.063 INFO 4486 --- [Check obproxy1,a8606a90e92f2f5f,a1d524d9d7401265] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:28:40.064 INFO 4486 --- [Collect obproxy2,fbb56d58860f2e46,63388d3b543444c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:28:40.070 INFO 4486 --- [Collect obproxy1,081357183666fae9,22a62a581ee7f7bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:28:40.076 INFO 4486 --- [Collect obproxy1,081357183666fae9,22a62a581ee7f7bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:28:40.082 INFO 4486 --- [Collect obproxy1,081357183666fae9,22a62a581ee7f7bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:28:40.128 INFO 4486 --- [ocp-schedules-18,4c60796ace592d7d,863aa4c332ec99cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:28:40.681 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,357f1296dff37be0,4314bb335b001052] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=357f1296dff37be0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:40.681 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,357f1296dff37be0,4314bb335b001052] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=357f1296dff37be0, duration=1 ms] 2024-10-16 13:28:40.994 INFO 4486 --- [prometheus-minute-schedule-1,ad444fdc7812d10a,d24af270e8cca990] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:40.995 INFO 4486 --- [prometheus-hour-schedule-1,d0928c0945189e19,5feae4ec36230d77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:28:40.995 INFO 4486 --- [ocp-async-8,a59ef1d482a0dc3c,e84c916b0c6ce153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:41.001 INFO 4486 --- [prometheus-minute-schedule-1,ad444fdc7812d10a,d24af270e8cca990] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:41.086 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e696ebb8a43920fe,b4df4d6c538c3ccf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e696ebb8a43920fe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:41.086 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e696ebb8a43920fe,b4df4d6c538c3ccf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e696ebb8a43920fe, duration=1 ms] 2024-10-16 13:28:42.067 INFO 4486 --- [ocp-schedules-26,75f4c9da3d4a328b,fd5e9dbf89e82bb3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:28:42.646 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:42.977 INFO 4486 --- [ocp-async-7,6ece07b1aedcf2b7,7b12b4eb9730d901] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:28:42.995 INFO 4486 --- [ocp-async-4,555bbb44de510686,e9519528a906d3b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:43.440 INFO 4486 --- [ocp-schedules-20,710692fccecc9eb8,c7ea1986b1fb2c9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:28:43.780 INFO 4486 --- [ocp-schedules-21,2cc08148d9a838c6,c6fcd3c26a54a32c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:28:43.850 WARN 4486 --- [scanner-13237318-1,2cc08148d9a838c6,be0598ef6e031f2a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:28:43.880 INFO 4486 --- [alarm-detect-9,2cc08148d9a838c6,e5f559760a7c9720] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:43.887 INFO 4486 --- [alarm-detect-9,2cc08148d9a838c6,e5f559760a7c9720] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:43.888 INFO 4486 --- [alarm-detect-5,2cc08148d9a838c6,6fb2843db17f43c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:43.895 INFO 4486 --- [alarm-detect-5,2cc08148d9a838c6,6fb2843db17f43c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:43.896 INFO 4486 --- [alarm-detect-9,2cc08148d9a838c6,e5f559760a7c9720] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:28:43.904 INFO 4486 --- [alarm-detect-5,2cc08148d9a838c6,6fb2843db17f43c6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:28:43.915 INFO 4486 --- [alarm-detect-15,2cc08148d9a838c6,817c564d49bd2d4f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:43.921 INFO 4486 --- [alarm-detect-15,2cc08148d9a838c6,817c564d49bd2d4f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:43.926 INFO 4486 --- [alarm-detect-15,2cc08148d9a838c6,817c564d49bd2d4f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:28:43.950 INFO 4486 --- [alarm-detect-4,2cc08148d9a838c6,659da538d42aa71c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056495 ~ 1729056513 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056510, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056510, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056510=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:28:43.994 INFO 4486 --- [prometheus-minute-schedule-1,ddc8925485af54a3,db028553a8dc5a40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:44.009 INFO 4486 --- [prometheus-minute-schedule-1,ddc8925485af54a3,db028553a8dc5a40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:44.027 INFO 4486 --- [alarm-detect-5,2cc08148d9a838c6,b07cd57d5c8986d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:44.034 INFO 4486 --- [alarm-detect-5,2cc08148d9a838c6,b07cd57d5c8986d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:44.040 INFO 4486 --- [alarm-detect-5,2cc08148d9a838c6,b07cd57d5c8986d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:28:44.995 INFO 4486 --- [ocp-async-3,dcfb83e45e174b5e,e7598c0e6af14d3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:45.415 INFO 4486 --- [ocp-schedules-24,96c1582a1491e206,51ec2001eecd942a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:28:45.420 INFO 4486 --- [ocp-schedules-24,96c1582a1491e206,51ec2001eecd942a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:28:45.420 INFO 4486 --- [ocp-schedules-24,96c1582a1491e206,51ec2001eecd942a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:28:45.429 INFO 4486 --- [ocp-schedules-6,16f80f05c5e4b1df,584ff4dc285b6036] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:28:45.429 INFO 4486 --- [ocp-schedules-6,16f80f05c5e4b1df,584ff4dc285b6036] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:28:45.647 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:46.853 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,97c182c7a3afb2ff,fd68fc7930fc8ae5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=97c182c7a3afb2ff, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:46.854 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,97c182c7a3afb2ff,fd68fc7930fc8ae5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=97c182c7a3afb2ff, duration=2 ms] 2024-10-16 13:28:46.994 INFO 4486 --- [prometheus-minute-schedule-1,38a23b5ada3c90cb,589010e6d613f323] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:46.995 INFO 4486 --- [ocp-async-10,d20a2e86cc1c43f9,6a90cbb881a0a1aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:47.000 INFO 4486 --- [prometheus-minute-schedule-1,38a23b5ada3c90cb,589010e6d613f323] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:47.052 INFO 4486 --- [ocp-schedules-5,e73133a120326db5,a379f4c7a733a4a7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:28:47.070 INFO 4486 --- [ocp-schedules-11,960c79d7ea11e005,c2a16b613bbb7a06] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:28:47.196 INFO 4486 --- [ocp-schedules-27,37f1de15bf77fe0f,4a6369d4bd3bdcb2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:28:48.647 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:48.996 INFO 4486 --- [ocp-async-5,7457464612852c89,13235cfb6b278da1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:49.063 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,37f8b2c1bc35a364,3d7828779eae45a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=37f8b2c1bc35a364, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:49.063 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,37f8b2c1bc35a364,3d7828779eae45a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=37f8b2c1bc35a364, duration=1 ms] 2024-10-16 13:28:49.607 INFO 4486 --- [ocp-schedules-7,686ec55b8653c660,7f76dcf80bfeeb1f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:28:49.607 INFO 4486 --- [ocp-schedules-7,686ec55b8653c660,7f76dcf80bfeeb1f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:28:49.607 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:28:49.607 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:28:49.607 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:28:49.607 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:28:49.607 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:28:49.616 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:28:49.616 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:28:49.617 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:49.617 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:28:49.617 INFO 4486 --- [ocp-schedules-7,686ec55b8653c660,7f76dcf80bfeeb1f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:28:49.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:28:49.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:28:49.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:49.619 INFO 4486 --- [ocp-schedules-7,686ec55b8653c660,7f76dcf80bfeeb1f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:28:49.619 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:28:49.619 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:28:49.619 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:28:49.649 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:28:49.650 INFO 4486 --- [ocp-schedules-7,686ec55b8653c660,7f76dcf80bfeeb1f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:28:49.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:28:49.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:28:49.654 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:28:49.654 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:28:49.657 INFO 4486 --- [ocp-schedules-7,686ec55b8653c660,7f76dcf80bfeeb1f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:28:49.994 INFO 4486 --- [prometheus-minute-schedule-1,e8eb706769afc8ed,7b1dca3e34232c16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:49.999 INFO 4486 --- [prometheus-minute-schedule-1,e8eb706769afc8ed,7b1dca3e34232c16] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:50.134 INFO 4486 --- [ocp-schedules-5,08d1190e28f99355,4af494eeeae55303] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:28:50.335 INFO 4486 --- [ocp-schedules-14,3fdc448651bf7b5d,810012283cfe8e0f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:28:50.335 INFO 4486 --- [ocp-schedules-14,3fdc448651bf7b5d,810012283cfe8e0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:28:50.337 INFO 4486 --- [ocp-schedules-14,3fdc448651bf7b5d,810012283cfe8e0f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:28:50.337 INFO 4486 --- [ocp-schedules-14,3fdc448651bf7b5d,810012283cfe8e0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:28:50.338 INFO 4486 --- [ocp-schedules-14,3fdc448651bf7b5d,810012283cfe8e0f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:28:50.338 INFO 4486 --- [ocp-schedules-14,3fdc448651bf7b5d,810012283cfe8e0f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:28:50.686 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d82cf1300aa3f202,3f7dfe49e255beae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d82cf1300aa3f202, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:50.686 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d82cf1300aa3f202,3f7dfe49e255beae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d82cf1300aa3f202, duration=2 ms] 2024-10-16 13:28:50.996 INFO 4486 --- [ocp-async-2,7849d3daea9e2013,9329e6267abd953c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:51.091 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4c816b1faf531443,b8d064e4b3c397e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4c816b1faf531443, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:51.092 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4c816b1faf531443,b8d064e4b3c397e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4c816b1faf531443, duration=1 ms] 2024-10-16 13:28:51.648 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:52.977 INFO 4486 --- [ocp-async-9,08ec002859b35250,45e4f012f3c45013] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:28:52.994 INFO 4486 --- [prometheus-minute-schedule-1,11e2e037127df8a6,be37723ad5e3d73d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:52.996 INFO 4486 --- [ocp-async-10,a695d2bad3c91bb5,ef3e0dbc85687131] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:53.000 INFO 4486 --- [prometheus-minute-schedule-1,11e2e037127df8a6,be37723ad5e3d73d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:54.058 INFO 4486 --- [ocp-schedules-16,a27434908d9f751a,4645ff58b1962fe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:28:54.058 INFO 4486 --- [ocp-schedules-16,99acfb36aa3e1bc9,dba14b468650409c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:28:54.060 INFO 4486 --- [ocp-schedules-16,99acfb36aa3e1bc9,dba14b468650409c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:54.072 INFO 4486 --- [ocp-schedules-8,bd98d077a990c813,429aba0896b94d7c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:28:54.095 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.096 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:54.116 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.118 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:54.130 WARN 4486 --- [scanner-13237613-1,bd98d077a990c813,9302625d4724252d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:28:54.162 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.163 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:54.184 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.188 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.189 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:28:54.260 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.261 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.261 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:54.262 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:28:54.263 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.265 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:28:54.268 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:54.278 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.278 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:54.288 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.297 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.298 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.299 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:54.300 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:28:54.305 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:28:54.309 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:28:54.311 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:28:54.334 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.334 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:28:54.336 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.337 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:28:54.347 INFO 4486 --- [ocp-schedules-3,1f9fce3525108204,0407c5da452944e7] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:28:54.347+08:00, tenantSize=5 2024-10-16 13:28:54.353 INFO 4486 --- [ocp-schedules-3,1f9fce3525108204,0407c5da452944e7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729056534347000 2024-10-16 13:28:54.353 INFO 4486 --- [ocp-schedules-3,1f9fce3525108204,0407c5da452944e7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729056234284000 2024-10-16 13:28:54.365 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:28:54.365 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729055940309000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:28:54.373 INFO 4486 --- [ocp-schedules-3,1f9fce3525108204,0407c5da452944e7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:28:54.380 INFO 4486 --- [ocp-schedules-3,1f9fce3525108204,0407c5da452944e7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729056534377000 2024-10-16 13:28:54.380 INFO 4486 --- [ocp-schedules-3,1f9fce3525108204,0407c5da452944e7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729056234308000 2024-10-16 13:28:54.393 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.394 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.395 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:28:54.399 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.403 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:28:54.406 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:54.412 INFO 4486 --- [ocp-schedules-3,1f9fce3525108204,0407c5da452944e7] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:28:54.421 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:28:54.430 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.431 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.433 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:28:54.433 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.433 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:54.445 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:28:54.445 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:28:54.465 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.465 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:54.494 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.494 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:54.531 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.531 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:28:54.558 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.559 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:28:54.586 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:28:54.613 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.615 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.616 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:28:54.620 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:28:54.624 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:28:54.628 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:28:54.630 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:28:54.649 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:54.657 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:28:54.793 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:28:54.823 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:28:54.824 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:28:54.825 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:28:54.858 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:28:54.858 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:28:54.996 INFO 4486 --- [ocp-async-6,3ad0b92a8892ed58,caf13980f34fdd52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:55.994 INFO 4486 --- [prometheus-minute-schedule-1,4819bae6d30b8a2d,c2be267267d20658] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:56.001 INFO 4486 --- [prometheus-minute-schedule-1,4819bae6d30b8a2d,c2be267267d20658] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:56.859 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,8a82f0bcf077621c,64326979a9ac5bdd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8a82f0bcf077621c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:56.860 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,8a82f0bcf077621c,64326979a9ac5bdd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8a82f0bcf077621c, duration=2 ms] 2024-10-16 13:28:56.997 INFO 4486 --- [ocp-async-3,a582a41255517cbb,4e1239ffd1e84694] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:57.144 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:28:57.144 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:28:57.152 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:28:57.152 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:28:57.152 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:28:57.152 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:28:57.160 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:28:57.160 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:28:57.160 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:28:57.160 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:28:57.209 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:28:57.217 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:28:57.217 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:28:57.217 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:28:57.217 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:28:57.217 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:28:57.224 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:28:57.650 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:28:57.684 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:28:58.206 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.046319753S 2024-10-16 13:28:58.206 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:28:58.285 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:28:58.285 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:28:58.313 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:28:58.397 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:28:58.397 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:28:58.397 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:28:58.397 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:28:58.397 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:28:58.398 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:28:58.407 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:28:58.535 INFO 4486 --- [ocp-schedules-24,3748e8236ccf375f,30c519731a29e773] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:28:58.537 INFO 4486 --- [ocp-schedules-24,3748e8236ccf375f,30c519731a29e773] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:28:58.585 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] o.s.boot.SpringApplication : Started application in 1.433 seconds (JVM running for 77441.636) 2024-10-16 13:28:58.585 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.433625147S 2024-10-16 13:28:58.790 INFO 4486 --- [ocp-schedules-20,cbc0e54b69575dd5,30682df810cda923] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:28:58.933 INFO 4486 --- [ocp-schedules-11,dfbaaf86a055494e,c5781750bce55b12] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:28:58.994 INFO 4486 --- [prometheus-minute-schedule-1,a867946467245bd3,60279e668bdbae9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056540, interval=60 2024-10-16 13:28:58.997 INFO 4486 --- [ocp-async-9,515117a629b07f82,095f6fae03915c30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:28:59.000 INFO 4486 --- [prometheus-minute-schedule-1,a867946467245bd3,60279e668bdbae9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:28:59.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c7c4e53c1850e32f,d1a6d0ed7de84d79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c7c4e53c1850e32f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:28:59.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c7c4e53c1850e32f,d1a6d0ed7de84d79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c7c4e53c1850e32f, duration=2 ms] 2024-10-16 13:29:00.140 INFO 4486 --- [ocp-schedules-20,bf59d7d5feed7f55,d09e10b595bff23a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:29:00.307 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5b2e84227409d67e,db8c6ba0d3a79340] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obce_3zones&version=2&ObClusterId=1701313639, client=10.197.49.176, traceId=5b2e84227409d67e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obce_3zones,1701313639,2,{"ObRegion":"obce_3zones","ObCluster":"obce_3zones","ObRegionId":1701313639,"ObClusterId":1701313639,"Type":"PRIMARY","timestamp":1729056540306992,"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:29:00.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5b2e84227409d67e,db8c6ba0d3a79340] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=obce_3zones, obClusterId=1701313639, rootServerJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, groupName=obce_3zones, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:29:00.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5b2e84227409d67e,db8c6ba0d3a79340] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:29:00.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5b2e84227409d67e,db8c6ba0d3a79340] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=obce_3zones, clusterId=1701313639, affectRows=1 2024-10-16 13:29:00.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5b2e84227409d67e,db8c6ba0d3a79340] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.176, traceId=5b2e84227409d67e, duration=8 ms] 2024-10-16 13:29:00.652 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:00.696 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,57ad99c71700fc48,daadd93406a5df0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=57ad99c71700fc48, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:00.697 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,57ad99c71700fc48,daadd93406a5df0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=57ad99c71700fc48, duration=8 ms] 2024-10-16 13:29:00.811 INFO 4486 --- [ocp-schedules-21,53f14089af3be7dd,c8223c0c004f3323] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:29:00.811+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:29:00.814 INFO 4486 --- [ocp-schedules-21,53f14089af3be7dd,c8223c0c004f3323] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:29:00.824 INFO 4486 --- [ocp-schedules-21,53f14089af3be7dd,c8223c0c004f3323] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:29:00.835 INFO 4486 --- [ocp-schedules-21,53f14089af3be7dd,c8223c0c004f3323] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:29:00.997 INFO 4486 --- [ocp-async-7,f8371e32b0c3de10,69faddb7cf8c03e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:01.098 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,274e442c66964776,f441e5704b97b66a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=274e442c66964776, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:01.098 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,274e442c66964776,f441e5704b97b66a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=274e442c66964776, duration=1 ms] 2024-10-16 13:29:01.994 INFO 4486 --- [prometheus-minute-schedule-1,f28bdd47f6fbae8f,de97e137c8e2ad5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:02.001 INFO 4486 --- [prometheus-minute-schedule-1,f28bdd47f6fbae8f,de97e137c8e2ad5b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:02.977 INFO 4486 --- [ocp-async-6,7219f4c99b10f385,168c3caf8d85b411] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:29:02.997 INFO 4486 --- [ocp-async-4,1c8c2df471c13e82,64cec15e9b8bc30d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:03.653 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:04.263 INFO 4486 --- [ocp-schedules-22,8ebf09c87e7082dc,90b8c3f7bdc577cb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:29:04.300 WARN 4486 --- [scanner-13237854-1,8ebf09c87e7082dc,55e307fe6245bd3b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:29:04.337 INFO 4486 --- [alarm-detect-15,8ebf09c87e7082dc,b851bd4606db0f73] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:04.342 INFO 4486 --- [alarm-detect-15,8ebf09c87e7082dc,b851bd4606db0f73] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:04.347 INFO 4486 --- [alarm-detect-15,8ebf09c87e7082dc,b851bd4606db0f73] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:04.350 INFO 4486 --- [alarm-detect-12,8ebf09c87e7082dc,7db2d8b829ffe4d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:04.350 INFO 4486 --- [alarm-detect-19,8ebf09c87e7082dc,8513d55636e717ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:04.355 INFO 4486 --- [alarm-detect-19,8ebf09c87e7082dc,8513d55636e717ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:04.355 INFO 4486 --- [alarm-detect-12,8ebf09c87e7082dc,7db2d8b829ffe4d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:04.359 INFO 4486 --- [alarm-detect-12,8ebf09c87e7082dc,7db2d8b829ffe4d1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:04.359 INFO 4486 --- [alarm-detect-19,8ebf09c87e7082dc,8513d55636e717ac] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:04.374 INFO 4486 --- [alarm-detect-5,8ebf09c87e7082dc,f7d2136414972a1d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056510 ~ 1729056534 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056525, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056525, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056525=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:29:04.436 INFO 4486 --- [alarm-detect-17,8ebf09c87e7082dc,0f8849960058b75d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:04.442 INFO 4486 --- [alarm-detect-17,8ebf09c87e7082dc,0f8849960058b75d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:04.446 INFO 4486 --- [alarm-detect-17,8ebf09c87e7082dc,0f8849960058b75d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:04.994 INFO 4486 --- [prometheus-minute-schedule-1,2c4ad4306ac3da07,a4e28e424a0a5dec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:04.997 INFO 4486 --- [ocp-async-1,678af4b6268b35a2,dc913b98b5c07a12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:05.001 INFO 4486 --- [prometheus-minute-schedule-1,2c4ad4306ac3da07,a4e28e424a0a5dec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:06.654 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:06.866 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9ba31d3cea03b807,43d71d1a4eb18bf4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9ba31d3cea03b807, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:06.866 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9ba31d3cea03b807,43d71d1a4eb18bf4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9ba31d3cea03b807, duration=2 ms] 2024-10-16 13:29:06.998 INFO 4486 --- [ocp-async-5,f283084cfe108dff,e3f09874134b8776] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:07.066 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:29:07.066 INFO 4486 --- [ocp-schedules-3,3cd8939cb6d1b035,2209a3227ba5bfc4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:29:07.068 INFO 4486 --- [ocp-schedules-3,3cd8939cb6d1b035,2209a3227ba5bfc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:29:07.081 INFO 4486 --- [ocp-schedules-7,70b2ecf01dbcd429,2cbe2bbcfe7d3aeb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:29:07.083 INFO 4486 --- [ocp-schedules-7,70b2ecf01dbcd429,2cbe2bbcfe7d3aeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:29:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:29:07.090 INFO 4486 --- [ocp-schedules-8,0bf373bdb9018a8b,c118e6486f3a74f3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:29:07.091 INFO 4486 --- [ocp-schedules-6,75b37ff8a8ed6b96,2d7cf81747931066] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:29:07.091 INFO 4486 --- [ocp-schedules-17,b9ecd9353aa07649,11913e228d1a239b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:29:07.091 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:29:07.091 INFO 4486 --- [host-check-agent-status1,75b37ff8a8ed6b96,809c36c50b7b3032] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:29:07.092 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:29:07.092 INFO 4486 --- [host-check-clockdiff1,b9ecd9353aa07649,75f5f174bef42744] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:29:07.093 INFO 4486 --- [host-check-agent-status3,75b37ff8a8ed6b96,8bcf7e3f7aa2a01b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:29:07.093 INFO 4486 --- [host-check-agent-status2,75b37ff8a8ed6b96,416e01e786ce0b48] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:29:07.093 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:29:07.094 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:29:07.095 INFO 4486 --- [host-check-clockdiff2,b9ecd9353aa07649,64ec066a0f46107b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:29:07.095 INFO 4486 --- [host-check-agent-status4,75b37ff8a8ed6b96,109552caa10430bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:29:07.095 INFO 4486 --- [host-check-clockdiff3,b9ecd9353aa07649,c4153ac10bf8457c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:29:07.096 INFO 4486 --- [host-check-clockdiff4,b9ecd9353aa07649,9ae7b2c303a2791d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:29:07.101 INFO 4486 --- [host-check-agent-status2,75b37ff8a8ed6b96,416e01e786ce0b48] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:29:07.101 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:07.101 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:29:07.101 INFO 4486 --- [host-check-agent-status3,75b37ff8a8ed6b96,8bcf7e3f7aa2a01b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:29:07.108 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:29:07.111 INFO 4486 --- [host-check-agent-status1,75b37ff8a8ed6b96,809c36c50b7b3032] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:07.115 INFO 4486 --- [host-check-agent-status3,75b37ff8a8ed6b96,8bcf7e3f7aa2a01b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:29:07.115 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:29:07.125 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:29:07.129 INFO 4486 --- [host-check-clockdiff1,b9ecd9353aa07649,75f5f174bef42744] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056547 0 0 , error= 2024-10-16 13:29:07.129 INFO 4486 --- [host-check-agent-status2,75b37ff8a8ed6b96,416e01e786ce0b48] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:29:07.129 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:07.130 INFO 4486 --- [host-check-agent-status4,75b37ff8a8ed6b96,109552caa10430bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:29:07.130 INFO 4486 --- [host-check-agent-status1,75b37ff8a8ed6b96,809c36c50b7b3032] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:07.131 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:29:07.132 INFO 4486 --- [host-check-agent-status4,75b37ff8a8ed6b96,109552caa10430bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:29:07.132 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:29:07.132 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.133 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:07.134 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:29:07.154 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.156 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:07.156 INFO 4486 --- [host-check-agent-status2,75b37ff8a8ed6b96,416e01e786ce0b48] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:29:07.156 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:29:07.158 INFO 4486 --- [host-check-agent-status3,75b37ff8a8ed6b96,8bcf7e3f7aa2a01b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:29:07.158 INFO 4486 --- [host-check-agent-status1,75b37ff8a8ed6b96,809c36c50b7b3032] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:29:07.158 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:29:07.159 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:29:07.159 INFO 4486 --- [host-check-agent-status4,75b37ff8a8ed6b96,109552caa10430bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:29:07.160 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:29:07.162576757+08:00 2024-10-16 13:29:07.160 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:29:07.160 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:29:07.160+08:00 2024-10-16 13:29:07.161 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:29:07.164178026+08:00 2024-10-16 13:29:07.161 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:29:07.161288220+08:00 2024-10-16 13:29:07.162 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:29:07.162 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:29:07.161+08:00 2024-10-16 13:29:07.162 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:29:07.161+08:00 2024-10-16 13:29:07.163 INFO 4486 --- [host-check-agent-status3,75b37ff8a8ed6b96,8bcf7e3f7aa2a01b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:29:07.163 INFO 4486 --- [host-check-agent-status3,75b37ff8a8ed6b96,8bcf7e3f7aa2a01b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:29:07.163 INFO 4486 --- [host-check-agent-status3,75b37ff8a8ed6b96,8bcf7e3f7aa2a01b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:29:07.164 INFO 4486 --- [host-check-agent-status1,75b37ff8a8ed6b96,809c36c50b7b3032] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:29:07.164 INFO 4486 --- [host-check-agent-status1,75b37ff8a8ed6b96,809c36c50b7b3032] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:29:07.164 INFO 4486 --- [host-check-agent-status1,75b37ff8a8ed6b96,809c36c50b7b3032] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:29:07.165 INFO 4486 --- [host-check-agent-status2,75b37ff8a8ed6b96,416e01e786ce0b48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:29:07.165 INFO 4486 --- [host-check-agent-status2,75b37ff8a8ed6b96,416e01e786ce0b48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:29:07.165 INFO 4486 --- [host-check-agent-status2,75b37ff8a8ed6b96,416e01e786ce0b48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:29:07.165 INFO 4486 --- [host-check-agent-status4,75b37ff8a8ed6b96,109552caa10430bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:29:07.165 INFO 4486 --- [host-check-agent-status4,75b37ff8a8ed6b96,109552caa10430bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:29:07.165 INFO 4486 --- [host-check-agent-status4,75b37ff8a8ed6b96,109552caa10430bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:29:07.166 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:29:07.168216456+08:00 2024-10-16 13:29:07.166 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:29:07.166+08:00 2024-10-16 13:29:07.173 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:29:07.160+08:00 and status AVAILABLE 2024-10-16 13:29:07.175 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:29:07.176 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:29:07.161+08:00 and status AVAILABLE 2024-10-16 13:29:07.177 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.177 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:29:07.161+08:00 and status AVAILABLE 2024-10-16 13:29:07.178 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:29:07.178 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:29:07.166+08:00 and status AVAILABLE 2024-10-16 13:29:07.180 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:29:07.180 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:29:07.181 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:29:07.184 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:29:07.186 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:29:07.188 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:29:07.189 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:07.189 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:29:07.189 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:29:07.190 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:07.191 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:29:07.195 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.196 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:29:07.206 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.207 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:29:07.216 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:07.217 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:07.219 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:07.220 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:07.222 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.223 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:29:07.225 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.226 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:29:07.236 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:07.238 INFO 4486 --- [ocp-schedules-29,bd41c58dee2d0157,5aa2f3ed35d431e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:29:07.246 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:29:07.257 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:29:07.259 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:29:07.262 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:29:07.268 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:29:07.271 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:29:07.285 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:07.285 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:29:07.286 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:07.288 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:07.296 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:29:07.311 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:07.313 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:29:07.321 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:29:07.324 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:29:07.332 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:29:07.333 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:29:07.340 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:07.344 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:29:07.350 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:07.361 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:07.370 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:29:07.370 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:29:07.380 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:29:07.381 INFO 4486 --- [host-check-host-status3,0bf373bdb9018a8b,b69e3354b33d6ca2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:29:07.387 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:29:07.396 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:07.397 INFO 4486 --- [host-check-host-status4,0bf373bdb9018a8b,3dd4ae3ac5a72cdd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:29:07.410 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:29:07.419 INFO 4486 --- [host-check-host-status2,0bf373bdb9018a8b,0f402ec6de0a9677] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:29:07.461 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:29:07.470 INFO 4486 --- [host-check-host-status1,0bf373bdb9018a8b,b908dd2c58a60e07] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:29:07.994 INFO 4486 --- [prometheus-minute-schedule-1,4acdb73b9440e9d3,ec1c6e81d9f21ada] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:08.000 INFO 4486 --- [prometheus-minute-schedule-1,4acdb73b9440e9d3,ec1c6e81d9f21ada] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:08.057 INFO 4486 --- [host-check-clockdiff3,b9ecd9353aa07649,c4153ac10bf8457c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056548 3 3 , error= 2024-10-16 13:29:08.567 INFO 4486 --- [ocp-schedules-24,3748e8236ccf375f,30c519731a29e773] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:29:08.901 INFO 4486 --- [host-check-clockdiff4,b9ecd9353aa07649,9ae7b2c303a2791d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056548 3 3 , error= 2024-10-16 13:29:08.998 INFO 4486 --- [ocp-async-6,2345c2759fad9cdc,4d054cde7b54f31b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:09.073 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0ace99b1f8f8ffed,043b6e5c5e8d4f8d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0ace99b1f8f8ffed, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:09.073 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0ace99b1f8f8ffed,043b6e5c5e8d4f8d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0ace99b1f8f8ffed, duration=1 ms] 2024-10-16 13:29:09.639 INFO 4486 --- [host-check-clockdiff2,b9ecd9353aa07649,64ec066a0f46107b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056549 3 3 , error= 2024-10-16 13:29:09.655 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:10.006 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:29:10.006 INFO 4486 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:29:10.027 INFO 4486 --- [Check obproxy2,da3ec376dd9d0184,8c78a2aa5a77b3ae] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:29:10.028 INFO 4486 --- [Check obproxy1,e038b56b8542d526,907ad85d044dc837] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:29:10.028 INFO 4486 --- [Collect obproxy2,3cb341ee507782ba,d549568d3b8cfc92] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:29:10.030 INFO 4486 --- [Collect obproxy1,32d98c0eeeb6e95b,abece795b9fd0a8d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:29:10.034 INFO 4486 --- [Collect obproxy2,3cb341ee507782ba,d549568d3b8cfc92] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:10.038 INFO 4486 --- [Check obproxy2,da3ec376dd9d0184,8c78a2aa5a77b3ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:10.043 INFO 4486 --- [Check obproxy1,e038b56b8542d526,907ad85d044dc837] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:10.046 INFO 4486 --- [Collect obproxy1,32d98c0eeeb6e95b,abece795b9fd0a8d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:10.057 INFO 4486 --- [Collect obproxy2,3cb341ee507782ba,d549568d3b8cfc92] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:29:10.063 INFO 4486 --- [Check obproxy2,da3ec376dd9d0184,8c78a2aa5a77b3ae] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:29:10.063 INFO 4486 --- [Check obproxy2,da3ec376dd9d0184,8c78a2aa5a77b3ae] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:29:10.063 INFO 4486 --- [Collect obproxy2,3cb341ee507782ba,d549568d3b8cfc92] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:29:10.068 INFO 4486 --- [Collect obproxy2,3cb341ee507782ba,d549568d3b8cfc92] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:29:10.070 INFO 4486 --- [Collect obproxy1,32d98c0eeeb6e95b,abece795b9fd0a8d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:29:10.073 INFO 4486 --- [Collect obproxy1,32d98c0eeeb6e95b,abece795b9fd0a8d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:29:10.075 INFO 4486 --- [Collect obproxy1,32d98c0eeeb6e95b,abece795b9fd0a8d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:29:10.086 INFO 4486 --- [Check obproxy1,e038b56b8542d526,907ad85d044dc837] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:29:10.086 INFO 4486 --- [Check obproxy1,e038b56b8542d526,907ad85d044dc837] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:29:10.147 INFO 4486 --- [ocp-schedules-19,59aca546955c76db,96227adc1dbc227f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:29:10.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f379c0e4791ac993,66868fac30bb023d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f379c0e4791ac993, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:10.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f379c0e4791ac993,66868fac30bb023d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f379c0e4791ac993, duration=1 ms] 2024-10-16 13:29:10.979 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:29:10.994 INFO 4486 --- [prometheus-minute-schedule-1,d1b0e1a3e456ca74,359e0c3bb1e151a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:10.998 INFO 4486 --- [ocp-async-8,1402b0ec0615ae65,7192008a83197289] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:11.000 INFO 4486 --- [prometheus-minute-schedule-1,d1b0e1a3e456ca74,359e0c3bb1e151a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:11.007 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:29:11.102 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,59038c2187672192,994341e807eddd23] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=59038c2187672192, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:11.102 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,59038c2187672192,994341e807eddd23] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=59038c2187672192, duration=1 ms] 2024-10-16 13:29:12.656 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:12.912 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=45, affectRows=45 2024-10-16 13:29:12.913 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=45, validCount=45 2024-10-16 13:29:12.915 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=81, affectRows=81 2024-10-16 13:29:12.915 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=81, validCount=81 2024-10-16 13:29:12.916 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=54, affectRows=54 2024-10-16 13:29:12.916 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=54, validCount=54 2024-10-16 13:29:12.919 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=92, affectRows=92 2024-10-16 13:29:12.919 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=92, validCount=92 2024-10-16 13:29:12.927 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=337, affectRows=337 2024-10-16 13:29:12.927 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=350, affectRows=350 2024-10-16 13:29:12.927 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=119, affectRows=119 2024-10-16 13:29:12.928 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=371, affectRows=371 2024-10-16 13:29:12.928 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=119, validCount=119 2024-10-16 13:29:12.928 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=337, validCount=337 2024-10-16 13:29:12.929 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=349, validCount=349 2024-10-16 13:29:12.930 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=371, validCount=371 2024-10-16 13:29:12.950 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=1000, affectRows=1000 2024-10-16 13:29:12.954 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=998, validCount=998 2024-10-16 13:29:12.960 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056480, nodeCount=1000, affectRows=1000 2024-10-16 13:29:12.965 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056480, nodeCount=1000, validCount=1000 2024-10-16 13:29:12.978 INFO 4486 --- [ocp-async-1,fa3bc60559ded63b,901e8cd78418534a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:29:12.998 INFO 4486 --- [ocp-async-7,f0901134a7bfa2ab,7e7ed4fbdcca63a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:13.994 INFO 4486 --- [prometheus-minute-schedule-1,c7bd12d6e7f04ef9,bc91a2c27d0aaeda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:14.000 INFO 4486 --- [prometheus-minute-schedule-1,c7bd12d6e7f04ef9,bc91a2c27d0aaeda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:14.318 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056480, nodeCount=480, affectRows=480 2024-10-16 13:29:14.508 INFO 4486 --- [ocp-schedules-23,efb114c9e7a08112,ee7db6c11f3b9e93] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:29:14.531 WARN 4486 --- [scanner-13238992-1,efb114c9e7a08112,861f1393c2f96078] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:29:14.587 INFO 4486 --- [alarm-detect-3,efb114c9e7a08112,5ab25c724a6785fe] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056525 ~ 1729056544 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056540, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056540, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056540=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:29:14.998 INFO 4486 --- [ocp-async-6,513ce4048d9de3af,de9c071140d01100] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:15.103 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056480, nodeCount=480, validCount=480 2024-10-16 13:29:15.657 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:16.874 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cc9b788587eaccde,4dbe62cac9cac8c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cc9b788587eaccde, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:16.874 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cc9b788587eaccde,4dbe62cac9cac8c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cc9b788587eaccde, duration=2 ms] 2024-10-16 13:29:16.994 INFO 4486 --- [prometheus-minute-schedule-1,e1ef4adbf25cc5be,6e378deb07bb4186] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:16.998 INFO 4486 --- [ocp-async-4,537ccf20b02f9ed7,428538211aca8353] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:17.000 INFO 4486 --- [prometheus-minute-schedule-1,e1ef4adbf25cc5be,6e378deb07bb4186] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:17.055 INFO 4486 --- [ocp-schedules-4,eef4419ac6b83501,f37fd296710c40cd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:29:17.069 INFO 4486 --- [ocp-schedules-3,3cd8939cb6d1b035,2209a3227ba5bfc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:29:17.161 INFO 4486 --- [ocp-schedules-7,70b2ecf01dbcd429,2cbe2bbcfe7d3aeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:29:18.659 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:18.795 INFO 4486 --- [ocp-schedules-1,cb08acfef4238271,2d839ab7ec915ff9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:29:18.999 INFO 4486 --- [ocp-async-10,8f0b9524496a602f,e546a899530629d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:19.079 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,3de7ffc6736a58b0,19ffddafda3e408c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3de7ffc6736a58b0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:19.079 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,3de7ffc6736a58b0,19ffddafda3e408c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3de7ffc6736a58b0, duration=2 ms] 2024-10-16 13:29:19.470 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ede5d682a185a278,06113551757036d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=ede5d682a185a278, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:29:19.471 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ede5d682a185a278,06113551757036d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=ede5d682a185a278, duration=2 ms] 2024-10-16 13:29:19.994 INFO 4486 --- [prometheus-minute-schedule-1,a9b0a9bc5b654ac1,1d1a3d1ddd7443a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:20.000 INFO 4486 --- [prometheus-minute-schedule-1,a9b0a9bc5b654ac1,1d1a3d1ddd7443a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:20.152 INFO 4486 --- [ocp-schedules-14,4f899542d1b9eaf2,dd8aaf3e3f7d804c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:29:20.708 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bcb7bc75c90e8ddf,bc00815d775dfcf2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bcb7bc75c90e8ddf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:20.708 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bcb7bc75c90e8ddf,bc00815d775dfcf2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bcb7bc75c90e8ddf, duration=1 ms] 2024-10-16 13:29:20.996 INFO 4486 --- [prometheus-cache-monitor1,61e1f4dd35ca6915,9c02119c5607a6ee] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13853,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:29:20.997 INFO 4486 --- [prometheus-cache-monitor1,61e1f4dd35ca6915,9c02119c5607a6ee] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:29:20.999 INFO 4486 --- [ocp-async-7,2b112056b177aedf,da9b6381d462e227] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:21.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,531023e6a66d23f2,b5569518de6140c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=531023e6a66d23f2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:21.107 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,531023e6a66d23f2,b5569518de6140c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=531023e6a66d23f2, duration=2 ms] 2024-10-16 13:29:21.660 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:22.581 INFO 4486 --- [ocp-schedules-2,d11878520d55d738,6a8b9ab1ecfe43b7] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:29:22.950 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d6a80f7bf983bd51,1108d7d3b830bc7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=d6a80f7bf983bd51, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:29:22.950 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d6a80f7bf983bd51,1108d7d3b830bc7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=d6a80f7bf983bd51, duration=1 ms] 2024-10-16 13:29:22.978 INFO 4486 --- [ocp-async-6,e4292c03b5b0191f,80d94af913057f85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:29:22.994 INFO 4486 --- [prometheus-minute-schedule-1,507e30eee52ab183,5d575fbeae963586] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:22.999 INFO 4486 --- [prometheus-minute-schedule-1,507e30eee52ab183,5d575fbeae963586] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:22.999 INFO 4486 --- [ocp-async-3,9ab4818b833db6e5,b14102c802b60642] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:23.396 INFO 4486 --- [ocp-schedules-12,9ac018f900a715f1,5fe6c37fef777c19] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:29:23.840 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4df52adc1d27f1e5,bff6942af9de0dbf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=4df52adc1d27f1e5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:29:23.840 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4df52adc1d27f1e5,bff6942af9de0dbf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=4df52adc1d27f1e5, duration=1 ms] 2024-10-16 13:29:24.039 INFO 4486 --- [ocp-schedules-28,51ce993073d52c12,d517b5246cff3cd6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:29:24.157 INFO 4486 --- [ocp-schedules-7,8e51cfcc18afd46d,5519c8f6bb0c6541] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:29:24.157 INFO 4486 --- [ocp-schedules-7,8e51cfcc18afd46d,5519c8f6bb0c6541] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:29:24.192 INFO 4486 --- [Collect all tenants major compactions1,8e51cfcc18afd46d,554dcdc901b29bd5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:29:24.192 INFO 4486 --- [Collect all tenants major compactions2,8e51cfcc18afd46d,293ca772fd9cc7f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:29:24.192 INFO 4486 --- [Collect all tenants major compactions4,8e51cfcc18afd46d,48e46162f421e08f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:29:24.193 INFO 4486 --- [Collect all tenants major compactions3,8e51cfcc18afd46d,9f0ed60e9fb567c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:29:24.193 INFO 4486 --- [Collect all tenants major compactions5,8e51cfcc18afd46d,962b1f5634727582] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:29:24.210 INFO 4486 --- [Collect all tenants major compactions2,8e51cfcc18afd46d,293ca772fd9cc7f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.211 INFO 4486 --- [Collect all tenants major compactions2,8e51cfcc18afd46d,293ca772fd9cc7f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:29:24.212 INFO 4486 --- [Collect all tenants major compactions3,8e51cfcc18afd46d,9f0ed60e9fb567c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.212 INFO 4486 --- [Collect all tenants major compactions1,8e51cfcc18afd46d,554dcdc901b29bd5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.214 INFO 4486 --- [Collect all tenants major compactions3,8e51cfcc18afd46d,9f0ed60e9fb567c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:29:24.214 INFO 4486 --- [Collect all tenants major compactions1,8e51cfcc18afd46d,554dcdc901b29bd5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:29:24.214 INFO 4486 --- [Collect all tenants major compactions2,8e51cfcc18afd46d,293ca772fd9cc7f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:29:24.217 INFO 4486 --- [Collect all tenants major compactions1,8e51cfcc18afd46d,554dcdc901b29bd5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:29:24.218 INFO 4486 --- [Collect all tenants major compactions3,8e51cfcc18afd46d,9f0ed60e9fb567c4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:29:24.219 INFO 4486 --- [Collect all tenants major compactions2,8e51cfcc18afd46d,293ca772fd9cc7f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:29:24.223 INFO 4486 --- [Collect all tenants major compactions3,8e51cfcc18afd46d,9f0ed60e9fb567c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:29:24.223 INFO 4486 --- [Collect all tenants major compactions5,8e51cfcc18afd46d,962b1f5634727582] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.224 INFO 4486 --- [Collect all tenants major compactions4,8e51cfcc18afd46d,48e46162f421e08f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.224 INFO 4486 --- [Collect all tenants major compactions1,8e51cfcc18afd46d,554dcdc901b29bd5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:29:24.225 INFO 4486 --- [Collect all tenants major compactions5,8e51cfcc18afd46d,962b1f5634727582] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:29:24.226 INFO 4486 --- [Collect all tenants major compactions4,8e51cfcc18afd46d,48e46162f421e08f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:29:24.229 INFO 4486 --- [Collect all tenants major compactions4,8e51cfcc18afd46d,48e46162f421e08f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:29:24.230 INFO 4486 --- [Collect all tenants major compactions4,8e51cfcc18afd46d,48e46162f421e08f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:29:24.231 INFO 4486 --- [Collect all tenants major compactions5,8e51cfcc18afd46d,962b1f5634727582] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:29:24.231 INFO 4486 --- [Collect all tenants major compactions5,8e51cfcc18afd46d,962b1f5634727582] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:29:24.232 INFO 4486 --- [ocp-schedules-7,8e51cfcc18afd46d,5519c8f6bb0c6541] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:29:24.662 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:24.662 INFO 4486 --- [ocp-schedules-23,8f37996f252785f9,2b3209924730f7dd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:29:24.712 WARN 4486 --- [scanner-13239241-1,8f37996f252785f9,13cbaa6837e8cb5d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:29:24.726 INFO 4486 --- [alarm-detect-12,8f37996f252785f9,66f0045d7930401b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:24.731 INFO 4486 --- [alarm-detect-12,8f37996f252785f9,66f0045d7930401b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:24.735 INFO 4486 --- [alarm-detect-12,8f37996f252785f9,66f0045d7930401b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:24.746 INFO 4486 --- [alarm-detect-17,8f37996f252785f9,99d764a75e6679fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:24.751 INFO 4486 --- [alarm-detect-10,8f37996f252785f9,9e8c94b930914e8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:24.752 INFO 4486 --- [alarm-detect-17,8f37996f252785f9,99d764a75e6679fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:24.756 INFO 4486 --- [alarm-detect-17,8f37996f252785f9,99d764a75e6679fa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:24.757 INFO 4486 --- [alarm-detect-10,8f37996f252785f9,9e8c94b930914e8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:24.762 INFO 4486 --- [alarm-detect-10,8f37996f252785f9,9e8c94b930914e8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:24.825 INFO 4486 --- [alarm-detect-4,8f37996f252785f9,5eb981cd25bbd872] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:24.830 INFO 4486 --- [alarm-detect-4,8f37996f252785f9,5eb981cd25bbd872] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:24.837 INFO 4486 --- [alarm-detect-4,8f37996f252785f9,5eb981cd25bbd872] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:24.858 INFO 4486 --- [ocp-schedules-16,99acfb36aa3e1bc9,dba14b468650409c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:29:24.859 INFO 4486 --- [ocp-schedules-16,431d5928838dd413,9b68636b97b4b5d0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:29:24.861 INFO 4486 --- [ocp-schedules-16,431d5928838dd413,9b68636b97b4b5d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:29:24.895 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.895 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:24.898 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.899 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:24.926 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.926 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:24.942 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:24.952 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.953 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:29:24.990 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:24.991 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.991 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:24.992 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:29:24.999 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:24.999 INFO 4486 --- [ocp-async-1,a6a1fcbf0b4009a5,885f948ab964fc0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:25.002 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:29:25.003 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:25.020 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.020 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:25.034 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.044 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:25.044 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:25.045 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:25.046 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:29:25.048 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:25.050 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:25.051 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:29:25.051 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:29:25.055 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:25.057 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:25.074 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:29:25.074 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:29:25.082 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:29:25.107 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:25.108 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:25.109 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:25.113 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:25.116 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:29:25.119 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:25.149 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.150 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:25.153 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:29:25.164 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:25.171 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:25.172 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:29:25.179 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.179 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:25.183 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:29:25.183 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:29:25.206 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.206 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:25.237 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.237 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:25.263 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.264 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:25.292 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:25.317 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:25.319 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:25.320 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:25.324 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:29:25.329 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:29:25.334 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:29:25.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:29:25.358 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:29:25.492 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:29:25.520 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:25.521 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:25.522 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:29:25.551 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:29:25.551 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:29:25.994 INFO 4486 --- [prometheus-minute-schedule-1,e97dbecc3842b90d,83fdcdab7f02e20e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:26.001 INFO 4486 --- [prometheus-minute-schedule-1,e97dbecc3842b90d,83fdcdab7f02e20e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:26.710 INFO 4486 --- [ocp-schedules-30,281641a899c8af14,1450a2cc35f68f5b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:29:26.710+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:29:26.712 INFO 4486 --- [ocp-schedules-30,281641a899c8af14,1450a2cc35f68f5b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:29:26.712 INFO 4486 --- [ocp-schedules-30,281641a899c8af14,1450a2cc35f68f5b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:29:26.713 INFO 4486 --- [ocp-schedules-30,281641a899c8af14,1450a2cc35f68f5b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:29:26.879 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,4475267724a4e224,dab1dc43bc9be95a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4475267724a4e224, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:26.879 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,4475267724a4e224,dab1dc43bc9be95a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4475267724a4e224, duration=1 ms] 2024-10-16 13:29:27.000 INFO 4486 --- [ocp-async-7,8a6727f6f5888c6c,67924c5d2b430dc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:27.663 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:28.994 INFO 4486 --- [prometheus-minute-schedule-1,65008956de0ad0d4,59386b21c47d9376] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:29.000 INFO 4486 --- [ocp-async-8,9162f087749050fa,dbb01bcd3a9b5fc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:29.000 INFO 4486 --- [prometheus-minute-schedule-1,65008956de0ad0d4,59386b21c47d9376] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:29.086 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,95e86e6c6fd3186f,553146c5578ec229] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=95e86e6c6fd3186f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:29.086 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,95e86e6c6fd3186f,553146c5578ec229] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=95e86e6c6fd3186f, duration=1 ms] 2024-10-16 13:29:30.103 INFO 4486 --- [prometheus-second-schedule-1,b1aafa165136b187,17f1482cb94f1a97] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:29:30.159 INFO 4486 --- [ocp-schedules-15,a920b8dfd1f33c56,ff5a1eb2a777465c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:29:30.664 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:30.712 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7985f50a8b0b396e,62ffac9b417c3e70] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7985f50a8b0b396e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:30.712 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7985f50a8b0b396e,62ffac9b417c3e70] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7985f50a8b0b396e, duration=1 ms] 2024-10-16 13:29:31.000 INFO 4486 --- [ocp-async-9,94385108b3547bfb,191704ed816525da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:31.111 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ac80c4129e296d0b,f135f3b9bec5ba25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ac80c4129e296d0b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:31.111 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ac80c4129e296d0b,f135f3b9bec5ba25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ac80c4129e296d0b, duration=1 ms] 2024-10-16 13:29:31.994 INFO 4486 --- [prometheus-minute-schedule-1,40bab9dde16b05a1,9c981f33baeff7cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:32.000 INFO 4486 --- [prometheus-minute-schedule-1,40bab9dde16b05a1,9c981f33baeff7cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:32.978 INFO 4486 --- [ocp-async-1,ae79713317b51048,bf517b2bfd2994b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:29:33.000 INFO 4486 --- [ocp-async-7,8ba5aba0187b1f3b,f214fc84f5ed7640] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:33.665 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:34.865 INFO 4486 --- [ocp-schedules-6,30f13a5ccd1c2d88,48266eba180ba007] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:29:34.889 WARN 4486 --- [scanner-13239551-1,30f13a5ccd1c2d88,09c0b495ec62125e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:29:34.932 INFO 4486 --- [alarm-detect-14,30f13a5ccd1c2d88,5c2798dfda213f89] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056540 ~ 1729056564 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056555, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056555, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056555=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:29:34.994 INFO 4486 --- [prometheus-minute-schedule-1,d3731ffc23fc0db1,1ecf28bb1a35fbcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:35.000 INFO 4486 --- [ocp-async-6,f2ce9b334dd0f4ea,d8a8a8af858509fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:35.004 INFO 4486 --- [prometheus-minute-schedule-1,d3731ffc23fc0db1,1ecf28bb1a35fbcc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:36.667 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:36.885 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1627e9cba134734e,dd01dd1c1552921b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1627e9cba134734e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:36.886 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1627e9cba134734e,dd01dd1c1552921b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1627e9cba134734e, duration=3 ms] 2024-10-16 13:29:37.000 INFO 4486 --- [ocp-async-4,4a57da661e1ed0a7,a6182e2bc31c26aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:37.063 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:29:37.064 INFO 4486 --- [ocp-schedules-14,4fc7eb1c8a3a6233,2d8c890936114b6c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:29:37.066 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:29:37.067 INFO 4486 --- [ocp-schedules-3,d2b7ffe05728e10a,6384e1ce6dcb65d9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:29:37.069 INFO 4486 --- [ocp-schedules-3,d2b7ffe05728e10a,6384e1ce6dcb65d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:29:37.069 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:29:37.071 INFO 4486 --- [ocp-schedules-14,4fc7eb1c8a3a6233,2d8c890936114b6c] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:29:37.081 INFO 4486 --- [ocp-schedules-5,cbbe4c96c9c8d94b,172d76e7c6ebfd05] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:29:37.083 INFO 4486 --- [ocp-schedules-5,cbbe4c96c9c8d94b,172d76e7c6ebfd05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:29:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:29:37.090 INFO 4486 --- [ocp-schedules-29,3e1627718bb2ac4e,5daefedd7fa7adc5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:29:37.090 INFO 4486 --- [ocp-schedules-11,39312ce63a3b580b,4ebbd48bc578b936] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:29:37.091 INFO 4486 --- [host-check-agent-status2,3e1627718bb2ac4e,676058e89b1e158d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:29:37.091 INFO 4486 --- [host-check-agent-status1,3e1627718bb2ac4e,3161cae27b7144d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:29:37.091 INFO 4486 --- [host-check-clockdiff1,39312ce63a3b580b,adc6c9968bdee3d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:29:37.091 INFO 4486 --- [ocp-schedules-9,87355eedb570a5e2,792c5a16a187a86e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:29:37.092 INFO 4486 --- [host-check-agent-status3,3e1627718bb2ac4e,17fd3c47d75e7a74] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:29:37.092 INFO 4486 --- [host-check-clockdiff2,39312ce63a3b580b,8201992dafbe1e54] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:29:37.093 INFO 4486 --- [host-check-clockdiff3,39312ce63a3b580b,96851bd614dac774] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:29:37.093 INFO 4486 --- [host-check-clockdiff4,39312ce63a3b580b,1f7a26df9db0d4f4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:29:37.094 INFO 4486 --- [host-check-agent-status4,3e1627718bb2ac4e,8965f34dbe71ead9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:29:37.094 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:29:37.094 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:29:37.094 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:29:37.094 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:29:37.101 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:29:37.101 INFO 4486 --- [host-check-agent-status2,3e1627718bb2ac4e,676058e89b1e158d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:29:37.102 INFO 4486 --- [host-check-agent-status3,3e1627718bb2ac4e,17fd3c47d75e7a74] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:29:37.102 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:37.101 INFO 4486 --- [host-check-agent-status1,3e1627718bb2ac4e,3161cae27b7144d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:37.101 INFO 4486 --- [host-check-agent-status4,3e1627718bb2ac4e,8965f34dbe71ead9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:29:37.118 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:29:37.119 INFO 4486 --- [host-check-agent-status2,3e1627718bb2ac4e,676058e89b1e158d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:29:37.119 INFO 4486 --- [host-check-agent-status3,3e1627718bb2ac4e,17fd3c47d75e7a74] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:29:37.119 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:29:37.131 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:29:37.131 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:29:37.131 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:37.131 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:37.136 INFO 4486 --- [host-check-agent-status4,3e1627718bb2ac4e,8965f34dbe71ead9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:29:37.136 INFO 4486 --- [host-check-clockdiff1,39312ce63a3b580b,adc6c9968bdee3d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056577 0 0 , error= 2024-10-16 13:29:37.136 INFO 4486 --- [host-check-agent-status1,3e1627718bb2ac4e,3161cae27b7144d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:37.136 INFO 4486 --- [host-check-clockdiff2,39312ce63a3b580b,8201992dafbe1e54] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056577 3 3 , error= 2024-10-16 13:29:37.143 INFO 4486 --- [host-check-clockdiff4,39312ce63a3b580b,1f7a26df9db0d4f4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056577 3 3 , error= 2024-10-16 13:29:37.143 INFO 4486 --- [host-check-clockdiff3,39312ce63a3b580b,96851bd614dac774] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056577 3 3 , error= 2024-10-16 13:29:37.149 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:29:37.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:37.188 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:29:37.208 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:29:37.208 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:37.210 INFO 4486 --- [host-check-agent-status2,3e1627718bb2ac4e,676058e89b1e158d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:29:37.211 INFO 4486 --- [host-check-agent-status1,3e1627718bb2ac4e,3161cae27b7144d8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:29:37.211 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:29:37.213484839+08:00 2024-10-16 13:29:37.211 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.211 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:29:37.211 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:29:37.211+08:00 2024-10-16 13:29:37.212 INFO 4486 --- [host-check-agent-status3,3e1627718bb2ac4e,17fd3c47d75e7a74] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:29:37.212 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:29:37.212 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:37.214 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:29:37.213869322+08:00 2024-10-16 13:29:37.214 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:29:37.211+08:00 and status AVAILABLE 2024-10-16 13:29:37.214 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.214 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:29:37.217061572+08:00 2024-10-16 13:29:37.215 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:29:37.214+08:00 2024-10-16 13:29:37.215 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:29:37.214+08:00 2024-10-16 13:29:37.216 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:29:37.216 INFO 4486 --- [host-check-agent-status3,3e1627718bb2ac4e,17fd3c47d75e7a74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:29:37.216 INFO 4486 --- [host-check-agent-status3,3e1627718bb2ac4e,17fd3c47d75e7a74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:29:37.216 INFO 4486 --- [host-check-agent-status3,3e1627718bb2ac4e,17fd3c47d75e7a74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:29:37.216 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.216 INFO 4486 --- [host-check-agent-status4,3e1627718bb2ac4e,8965f34dbe71ead9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:29:37.216 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:29:37.216 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-agent-status1,3e1627718bb2ac4e,3161cae27b7144d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-agent-status1,3e1627718bb2ac4e,3161cae27b7144d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-agent-status1,3e1627718bb2ac4e,3161cae27b7144d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-agent-status2,3e1627718bb2ac4e,676058e89b1e158d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-agent-status2,3e1627718bb2ac4e,676058e89b1e158d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-agent-status2,3e1627718bb2ac4e,676058e89b1e158d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:29:37.220290719+08:00 2024-10-16 13:29:37.217 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:29:37.217 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:29:37.214+08:00 and status AVAILABLE 2024-10-16 13:29:37.218 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:29:37.217+08:00 2024-10-16 13:29:37.218 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:29:37.214+08:00 and status AVAILABLE 2024-10-16 13:29:37.218 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.219 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.220 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:29:37.220 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:29:37.220 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:29:37.221 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:29:37.221 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:29:37.217+08:00 and status AVAILABLE 2024-10-16 13:29:37.223 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:29:37.223 INFO 4486 --- [host-check-agent-status4,3e1627718bb2ac4e,8965f34dbe71ead9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:29:37.223 INFO 4486 --- [host-check-agent-status4,3e1627718bb2ac4e,8965f34dbe71ead9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:29:37.223 INFO 4486 --- [host-check-agent-status4,3e1627718bb2ac4e,8965f34dbe71ead9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:29:37.227 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:29:37.229 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.230 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:29:37.230 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:29:37.230 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:29:37.231 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:29:37.231 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.231 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.231 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.231 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.231 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.232 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.232 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:29:37.232 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:37.234 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:29:37.234 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:37.239 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.240 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:29:37.249 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.250 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:29:37.250 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:29:37.252 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.253 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:29:37.253 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:37.259 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:37.261 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:37.266 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:29:37.274 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.275 INFO 4486 --- [ocp-schedules-25,bd2f7cd649298759,def80daa66bd98e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:29:37.283 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:29:37.294 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:29:37.299 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:29:37.299 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:29:37.311 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:29:37.312 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:37.314 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:29:37.316 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:29:37.326 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:29:37.330 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:37.333 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:37.341 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:29:37.342 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:29:37.351 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:29:37.364 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:29:37.366 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:29:37.367 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:37.374 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:29:37.378 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:29:37.392 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:37.394 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:29:37.395 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:37.396 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:29:37.405 INFO 4486 --- [host-check-host-status3,87355eedb570a5e2,37020483f54e503f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:29:37.405 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:29:37.420 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:29:37.433 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:29:37.436 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:29:37.443 INFO 4486 --- [host-check-host-status4,87355eedb570a5e2,2f1c266b593cbd80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:29:37.446 INFO 4486 --- [host-check-host-status2,87355eedb570a5e2,12f8aa22900544c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:29:37.479 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:37.480 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:29:37.481 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.482 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:29:37.483 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.484 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.485 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:29:37.491 INFO 4486 --- [host-check-host-status1,87355eedb570a5e2,e0380498f8cac57d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:29:37.496 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.496 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.496 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.496 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.496 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.497 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.525 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:29:37.565 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:29:37.609 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:37.625 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:37.626 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.627 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:29:37.628 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:29:37.633 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.634 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:29:37.635 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.636 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.637 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:29:37.648 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.648 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.648 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.648 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.648 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.649 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.656 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:29:37.851 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:29:37.893 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:37.895 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.896 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:29:37.897 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.897 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.898 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:29:37.909 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.909 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.909 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.909 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.909 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:37.909 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:37.927 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:29:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:29:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:29:37.994 INFO 4486 --- [prometheus-minute-schedule-1,25fa623ec113b988,12b80b69682d3ce3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:37.999 INFO 4486 --- [prometheus-minute-schedule-1,25fa623ec113b988,12b80b69682d3ce3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:38.126 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:29:38.170 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:38.173 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:38.174 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:29:38.175 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:38.175 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:38.176 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:29:38.187 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:38.187 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:38.187 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:38.187 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:38.187 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:29:38.188 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:38.255 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:29:38.430 INFO 4486 --- [ocp-schedules-19,d1207f7297e09fee,01fe63fc9c931365] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:29:38.430 INFO 4486 --- [ocp-schedules-19,d1207f7297e09fee,01fe63fc9c931365] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:29:38.431 INFO 4486 --- [ocp-schedules-19,d1207f7297e09fee,01fe63fc9c931365] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:29:38.450 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:29:38.258344, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:29:38.451 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:29:38.258344, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:29:38.464 INFO 4486 --- [ocp-schedules-25,72072b8dc35a2839,be3f11e20e360e78] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13853,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:29:38.464 INFO 4486 --- [ocp-schedules-25,72072b8dc35a2839,be3f11e20e360e78] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:29:38.493 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:29:38.258344} } 2024-10-16 13:29:38.557 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:29:38.563 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:29:38.567 INFO 4486 --- [ocp-schedules-12,b9ee8c4422201ad2,9bc7b819dc78420d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:29:38.800 INFO 4486 --- [ocp-schedules-18,20cb2085be5445f7,d94640353b4ed103] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:29:39.001 INFO 4486 --- [ocp-async-2,3f1ae239cdb923db,341ef491a8b0a62e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:39.090 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,42250fcd8b3d3f44,0cc5c09772a46d67] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=42250fcd8b3d3f44, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:39.090 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,42250fcd8b3d3f44,0cc5c09772a46d67] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=42250fcd8b3d3f44, duration=1 ms] 2024-10-16 13:29:39.668 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:40.006 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:29:40.006 INFO 4486 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:29:40.024 INFO 4486 --- [Check obproxy1,b71b4763f3f299eb,0dc9db1884e149dc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:29:40.024 INFO 4486 --- [Check obproxy1,b71b4763f3f299eb,0dc9db1884e149dc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:40.025 INFO 4486 --- [Check obproxy2,2d684b3d8fe019d6,2f2c6958aadb34e2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:29:40.025 INFO 4486 --- [Check obproxy2,2d684b3d8fe019d6,2f2c6958aadb34e2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:40.030 INFO 4486 --- [Collect obproxy2,cd91600206925351,eca34118d5beadbe] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:29:40.030 INFO 4486 --- [Collect obproxy2,cd91600206925351,eca34118d5beadbe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:40.030 INFO 4486 --- [Collect obproxy1,d8e358730da30cfe,06d0459f27ed5d5e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:29:40.030 INFO 4486 --- [Collect obproxy1,d8e358730da30cfe,06d0459f27ed5d5e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:29:40.047 INFO 4486 --- [Check obproxy2,2d684b3d8fe019d6,2f2c6958aadb34e2] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:29:40.047 INFO 4486 --- [Check obproxy2,2d684b3d8fe019d6,2f2c6958aadb34e2] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:29:40.049 INFO 4486 --- [Check obproxy1,b71b4763f3f299eb,0dc9db1884e149dc] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:29:40.049 INFO 4486 --- [Check obproxy1,b71b4763f3f299eb,0dc9db1884e149dc] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:29:40.051 INFO 4486 --- [Collect obproxy2,cd91600206925351,eca34118d5beadbe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:29:40.055 INFO 4486 --- [Collect obproxy2,cd91600206925351,eca34118d5beadbe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:29:40.058 INFO 4486 --- [Collect obproxy1,d8e358730da30cfe,06d0459f27ed5d5e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:29:40.060 INFO 4486 --- [Collect obproxy2,cd91600206925351,eca34118d5beadbe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:29:40.063 INFO 4486 --- [Collect obproxy1,d8e358730da30cfe,06d0459f27ed5d5e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:29:40.068 INFO 4486 --- [Collect obproxy1,d8e358730da30cfe,06d0459f27ed5d5e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:29:40.164 INFO 4486 --- [ocp-schedules-18,894ef7f0ab7fdff1,5b01d85aa46a2bc5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:29:40.717 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0db21bc3869ae995,c264e34dee7aada6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0db21bc3869ae995, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:40.717 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0db21bc3869ae995,c264e34dee7aada6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0db21bc3869ae995, duration=1 ms] 2024-10-16 13:29:40.994 INFO 4486 --- [prometheus-minute-schedule-1,905235c34ba2a69d,da0d85a76f8086fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:40.995 INFO 4486 --- [prometheus-hour-schedule-1,f4f5b7a873a81357,2c1f4963e89cbec5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:29:40.999 INFO 4486 --- [prometheus-minute-schedule-1,905235c34ba2a69d,da0d85a76f8086fc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:41.001 INFO 4486 --- [ocp-async-3,eef52f3abafb55a7,996ffac1f31f504b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:41.115 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6316bb29b132c616,489e92f5e52e9ffd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6316bb29b132c616, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:41.115 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6316bb29b132c616,489e92f5e52e9ffd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6316bb29b132c616, duration=1 ms] 2024-10-16 13:29:42.071 INFO 4486 --- [ocp-schedules-19,8353497de45fe416,3204a0c181bdf2c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:29:42.669 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:42.978 INFO 4486 --- [ocp-async-9,7d908b604b727c7d,8f82d6d10235f85b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:29:43.001 INFO 4486 --- [ocp-async-5,573b559bd710b345,e624266d4eb45b82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:43.441 INFO 4486 --- [ocp-schedules-13,2956fcda238fbe9f,0d53fc03bad7f486] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:29:43.443 INFO 4486 --- [ocp-schedules-13,2956fcda238fbe9f,0d53fc03bad7f486] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:29:43.444 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:29:43.444 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:29:43.444 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:29:43.994 INFO 4486 --- [prometheus-minute-schedule-1,bf7d464c975b7ce4,34feca968f63a7dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:44.001 INFO 4486 --- [prometheus-minute-schedule-1,bf7d464c975b7ce4,34feca968f63a7dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:45.001 INFO 4486 --- [ocp-async-7,5c6b686f97513b3e,123a679b3d57f4ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:45.009 INFO 4486 --- [ocp-schedules-8,f6d4be85c13bc466,cfb132e91563d230] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:29:45.049 WARN 4486 --- [scanner-13239815-1,f6d4be85c13bc466,e3ff0237f0d75ed6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:29:45.127 INFO 4486 --- [alarm-detect-8,f6d4be85c13bc466,87045b837fa733e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:45.132 INFO 4486 --- [alarm-detect-8,f6d4be85c13bc466,87045b837fa733e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:45.138 INFO 4486 --- [alarm-detect-11,f6d4be85c13bc466,1c87879bd653648d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:45.139 INFO 4486 --- [alarm-detect-5,f6d4be85c13bc466,048a16af64f64cd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:45.141 INFO 4486 --- [alarm-detect-8,f6d4be85c13bc466,87045b837fa733e9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:29:45.143 INFO 4486 --- [alarm-detect-11,f6d4be85c13bc466,1c87879bd653648d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:45.145 INFO 4486 --- [alarm-detect-5,f6d4be85c13bc466,048a16af64f64cd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:45.148 INFO 4486 --- [alarm-detect-11,f6d4be85c13bc466,1c87879bd653648d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:29:45.149 INFO 4486 --- [alarm-detect-5,f6d4be85c13bc466,048a16af64f64cd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:29:45.204 INFO 4486 --- [alarm-detect-2,f6d4be85c13bc466,4bf5e5ec67d1d07b] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056555 ~ 1729056575 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056570, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056570, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056570=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:29:45.257 INFO 4486 --- [alarm-detect-17,f6d4be85c13bc466,4ced83ba19b80ee5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:45.262 INFO 4486 --- [alarm-detect-17,f6d4be85c13bc466,4ced83ba19b80ee5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:45.267 INFO 4486 --- [alarm-detect-17,f6d4be85c13bc466,4ced83ba19b80ee5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:29:45.420 INFO 4486 --- [ocp-schedules-14,912c7cf4ce495143,c1c1cfda265b58a0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:29:45.425 INFO 4486 --- [ocp-schedules-14,912c7cf4ce495143,c1c1cfda265b58a0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:29:45.425 INFO 4486 --- [ocp-schedules-14,912c7cf4ce495143,c1c1cfda265b58a0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:29:45.435 INFO 4486 --- [ocp-schedules-28,3d40210bfe10f48e,6256bb4ad891e02a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:29:45.435 INFO 4486 --- [ocp-schedules-28,3d40210bfe10f48e,6256bb4ad891e02a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:29:45.669 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:46.892 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c53889d1eb329183,2a8abf7cdf88a5ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c53889d1eb329183, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:46.892 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c53889d1eb329183,2a8abf7cdf88a5ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c53889d1eb329183, duration=1 ms] 2024-10-16 13:29:46.994 INFO 4486 --- [prometheus-minute-schedule-1,fe73084be3619718,20aa2d362d9a17af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:47.000 INFO 4486 --- [prometheus-minute-schedule-1,fe73084be3619718,20aa2d362d9a17af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:47.001 INFO 4486 --- [ocp-async-6,d9e9d9e9a9e0748e,e951a364f48e36c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:47.058 INFO 4486 --- [ocp-schedules-12,7a1397a204fdd9d2,0168b1889ab5b4b6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:29:47.071 INFO 4486 --- [ocp-schedules-3,d2b7ffe05728e10a,6384e1ce6dcb65d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:29:47.217 INFO 4486 --- [ocp-schedules-5,cbbe4c96c9c8d94b,172d76e7c6ebfd05] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:29:48.670 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:48.783 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:48.783 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:29:48.785 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:48.785 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:29:48.786 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:29:49.002 INFO 4486 --- [ocp-async-9,361873596f43e0a1,42b3d1c544214bc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:49.097 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dfcc31bc550b9cc0,1cc2167b89b0ccae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dfcc31bc550b9cc0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:49.097 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dfcc31bc550b9cc0,1cc2167b89b0ccae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dfcc31bc550b9cc0, duration=2 ms] 2024-10-16 13:29:49.666 INFO 4486 --- [ocp-schedules-9,dd0848b3f079fc41,52b56893f0442250] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:29:49.666 INFO 4486 --- [ocp-schedules-9,dd0848b3f079fc41,52b56893f0442250] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:29:49.666 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:29:49.666 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:29:49.667 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:29:49.667 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:29:49.667 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:29:49.672 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:29:49.673 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:49.673 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:29:49.673 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:29:49.674 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:49.674 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:29:49.674 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:29:49.675 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:29:49.675 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:29:49.676 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:29:49.676 INFO 4486 --- [ocp-schedules-9,dd0848b3f079fc41,52b56893f0442250] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:29:49.678 INFO 4486 --- [ocp-schedules-9,dd0848b3f079fc41,52b56893f0442250] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:29:49.698 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:29:49.698 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:29:49.699 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:29:49.700 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:29:49.700 INFO 4486 --- [ocp-schedules-9,dd0848b3f079fc41,52b56893f0442250] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:29:49.700 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:29:49.703 INFO 4486 --- [ocp-schedules-9,dd0848b3f079fc41,52b56893f0442250] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:29:49.994 INFO 4486 --- [prometheus-minute-schedule-1,83a0beee193a4268,fbe61d3042e0c311] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:50.001 INFO 4486 --- [prometheus-minute-schedule-1,83a0beee193a4268,fbe61d3042e0c311] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:50.169 INFO 4486 --- [ocp-schedules-12,beabc5ead8f28c3d,05852f564229aa18] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:29:50.339 INFO 4486 --- [ocp-schedules-23,98b8b4c31c14d768,8a4c19b8a7ebd6a7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:29:50.339 INFO 4486 --- [ocp-schedules-23,98b8b4c31c14d768,8a4c19b8a7ebd6a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:29:50.346 INFO 4486 --- [ocp-schedules-23,98b8b4c31c14d768,8a4c19b8a7ebd6a7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:29:50.346 INFO 4486 --- [ocp-schedules-23,98b8b4c31c14d768,8a4c19b8a7ebd6a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:29:50.349 INFO 4486 --- [ocp-schedules-23,98b8b4c31c14d768,8a4c19b8a7ebd6a7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:29:50.349 INFO 4486 --- [ocp-schedules-23,98b8b4c31c14d768,8a4c19b8a7ebd6a7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:29:50.724 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1fdbfbd54fa4cf42,e3de0229f1bcdc89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1fdbfbd54fa4cf42, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:50.724 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1fdbfbd54fa4cf42,e3de0229f1bcdc89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1fdbfbd54fa4cf42, duration=1 ms] 2024-10-16 13:29:51.002 INFO 4486 --- [ocp-async-1,cbded5bcd08ec2a3,463378ca9e306560] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:51.121 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e96d3574d9a7bda0,08e76eb3eb63d425] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e96d3574d9a7bda0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:51.121 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e96d3574d9a7bda0,08e76eb3eb63d425] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e96d3574d9a7bda0, duration=1 ms] 2024-10-16 13:29:51.671 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:52.979 INFO 4486 --- [ocp-async-8,4072385aa2e57eec,345e980a12de3b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:29:52.994 INFO 4486 --- [prometheus-minute-schedule-1,83a0f8e534ce41e9,3f7ee46ac7871a13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:52.999 INFO 4486 --- [prometheus-minute-schedule-1,83a0f8e534ce41e9,3f7ee46ac7871a13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:53.002 INFO 4486 --- [ocp-async-6,c109e2cc239e5984,833de908c6440c88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:53.445 INFO 4486 --- [ocp-schedules-13,2956fcda238fbe9f,0d53fc03bad7f486] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:29:54.672 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:55.002 INFO 4486 --- [ocp-async-10,9385d6d0bb23c5f3,64f3338290a2fd1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:55.302 INFO 4486 --- [ocp-schedules-8,4d07f1d6d4071502,10d4748d0a142e43] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:29:55.319 WARN 4486 --- [scanner-13240122-1,4d07f1d6d4071502,76025a1150c24dba] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:29:55.371 INFO 4486 --- [alarm-detect-15,4d07f1d6d4071502,8db2516a02722694] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056570 ~ 1729056585 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056585, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056585, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056585=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:29:55.551 INFO 4486 --- [ocp-schedules-16,431d5928838dd413,9b68636b97b4b5d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:29:55.551 INFO 4486 --- [ocp-schedules-16,b18c017d9cb2654c,001958d98e52459a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:29:55.553 INFO 4486 --- [ocp-schedules-16,b18c017d9cb2654c,001958d98e52459a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:29:55.585 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.585 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:55.590 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.591 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:55.616 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.617 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:55.622 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:55.623 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.623 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:29:55.651 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:55.652 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.652 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:55.653 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:29:55.654 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.656 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:29:55.657 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:55.666 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.666 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:55.678 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.688 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:55.688 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.689 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:55.690 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:29:55.695 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:29:55.699 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:29:55.701 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:29:55.722 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:55.724 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.725 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:29:55.725 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:29:55.761 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:29:55.761 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:29:55.782 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:55.783 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.784 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:29:55.788 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.791 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:29:55.794 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:55.798 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:29:55.808 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:55.809 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.810 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:29:55.820 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:29:55.820 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:29:55.821 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.822 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:55.854 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.854 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:55.878 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.879 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:55.903 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.903 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:29:55.931 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.931 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:29:55.956 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:29:55.979 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:55.981 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:55.982 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:29:55.985 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:29:55.989 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:29:55.993 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:29:55.994 INFO 4486 --- [prometheus-minute-schedule-1,a1c2adff67be39a1,683bfbfd13b2c33b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:55.995 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:29:56.000 INFO 4486 --- [prometheus-minute-schedule-1,a1c2adff67be39a1,683bfbfd13b2c33b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:56.021 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:29:56.155 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:29:56.182 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:29:56.183 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:29:56.184 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:29:56.213 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:29:56.213 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:29:56.899 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b695185d79226b50,0bb8fd2f82dced63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b695185d79226b50, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:56.899 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b695185d79226b50,0bb8fd2f82dced63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b695185d79226b50, duration=1 ms] 2024-10-16 13:29:57.002 INFO 4486 --- [ocp-async-1,bbe99cd98d124216,89f7ce0eb611faa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:57.673 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:29:58.805 INFO 4486 --- [ocp-schedules-22,8c6a465ee7a25db6,c892e238f049210f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:29:58.994 INFO 4486 --- [prometheus-minute-schedule-1,3b2b33367c59e864,58ac1c5fbbe22b7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056600, interval=60 2024-10-16 13:29:59.000 INFO 4486 --- [prometheus-minute-schedule-1,3b2b33367c59e864,58ac1c5fbbe22b7f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:29:59.003 INFO 4486 --- [ocp-async-8,18e99d19a63d7930,be1253f65ccab204] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:29:59.107 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8c5818b6c3f5c871,5d55003d2127443f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8c5818b6c3f5c871, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:29:59.107 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8c5818b6c3f5c871,5d55003d2127443f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8c5818b6c3f5c871, duration=1 ms] 2024-10-16 13:30:00.175 INFO 4486 --- [ocp-schedules-9,eecec10717aa109c,32f27b05f7b69df2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:30:00.674 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:00.733 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0ce19bb44daff11f,1a8eaaf671e09afb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0ce19bb44daff11f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:00.733 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0ce19bb44daff11f,1a8eaaf671e09afb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0ce19bb44daff11f, duration=1 ms] 2024-10-16 13:30:00.858 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fdc92735dfa94df0,556827af1765ffdc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=fdc92735dfa94df0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:30:00.859 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fdc92735dfa94df0,556827af1765ffdc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=fdc92735dfa94df0, duration=2 ms] 2024-10-16 13:30:00.865 INFO 4486 --- [ocp-schedules-20,8a4a651de9e3d973,5a8bb1f20cf59b3a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:30:00.865+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:30:00.868 INFO 4486 --- [ocp-schedules-20,8a4a651de9e3d973,5a8bb1f20cf59b3a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:30:00.873 INFO 4486 --- [ocp-schedules-20,8a4a651de9e3d973,5a8bb1f20cf59b3a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:30:00.885 INFO 4486 --- [ocp-schedules-20,8a4a651de9e3d973,5a8bb1f20cf59b3a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:30:01.003 INFO 4486 --- [ocp-async-4,b4344bc1bd290360,c8997017371fcb5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:01.125 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bb2ca952e4534e3e,b85beb445dba007f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bb2ca952e4534e3e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:01.125 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bb2ca952e4534e3e,b85beb445dba007f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bb2ca952e4534e3e, duration=1 ms] 2024-10-16 13:30:01.994 INFO 4486 --- [prometheus-minute-schedule-1,863f57b04e8e70c0,2e7e11a2db3eede4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:02.000 INFO 4486 --- [prometheus-minute-schedule-1,863f57b04e8e70c0,2e7e11a2db3eede4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:02.153 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,98f7ca0ca5a143b5,130c72c7b949eb49] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=98f7ca0ca5a143b5, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 13:30:02.340 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,98f7ca0ca5a143b5,130c72c7b949eb49] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=98f7ca0ca5a143b5, duration=218 ms] 2024-10-16 13:30:02.979 INFO 4486 --- [ocp-async-5,9ce7e1f965cfe88b,8bd310a9092d1e05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:30:03.003 INFO 4486 --- [ocp-async-2,345a1b848b5c60cd,7fc792c667951571] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:03.675 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:04.994 INFO 4486 --- [prometheus-minute-schedule-1,c7aa0faac8c53113,aa494cb809d6b878] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:04.999 INFO 4486 --- [prometheus-minute-schedule-1,c7aa0faac8c53113,aa494cb809d6b878] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:05.003 INFO 4486 --- [ocp-async-3,56bd38f6f032263c,957b7e0a8c46e7c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:05.224 INFO 4486 --- [subtask-coordinator3,791e68b70ade5cad,fab3583089d801f1] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7932261, operation:EXECUTE, state: RUNNING 2024-10-16 13:30:05.237 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7932261, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266488, task_operation=execute, latest_execution_start_time=2024-10-16T13:30:05.218+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=7932261}, listMap={}}, executor=10.197.49.184 2024-10-16 13:30:05.242 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 13:30:05.243 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 13:30:05.245 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.245 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:30:05.247 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.248 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.249 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.249 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 13:30:05.268 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.269 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.270 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.270 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 13:30:05.284 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.284 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.285 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.285 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 13:30:05.299 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.300 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.300 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.301 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 13:30:05.314 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.315 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.315 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.316 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 13:30:05.322 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.323 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.324 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.324 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 13:30:05.330 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.330 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.331 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.332 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 13:30:05.338 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.339 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.339 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.340 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 13:30:05.347 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.347 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.348 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.348 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 13:30:05.355 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.355 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.356 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.356 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 13:30:05.362 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.363 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.363 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.364 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 13:30:05.373 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.373 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.374 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.374 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 13:30:05.388 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.388 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.389 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.389 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 13:30:05.394 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.395 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.396 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.396 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 13:30:05.402 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.402 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.403 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.404 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 13:30:05.409 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.410 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.411 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.411 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 13:30:05.416 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.416 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.417 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.417 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 13:30:05.421 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.422 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.422 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.423 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 13:30:05.426 INFO 4486 --- [ocp-schedules-15,197e3476c6f0d052,b5162a8d613524f3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:30:05.427 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.428 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.429 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.429 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 13:30:05.435 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.436 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.437 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.437 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 13:30:05.443 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.443 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.444 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.445 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 13:30:05.451 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.451 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.452 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.453 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 13:30:05.459 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.461 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.462 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.463 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 13:30:05.464 WARN 4486 --- [scanner-13240375-1,197e3476c6f0d052,076ec07f5b3c61d5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:30:05.470 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.471 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.472 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.472 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 13:30:05.476 INFO 4486 --- [alarm-detect-18,197e3476c6f0d052,d3807d54f8d416c1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:05.479 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.480 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.481 INFO 4486 --- [alarm-detect-18,197e3476c6f0d052,d3807d54f8d416c1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:05.481 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.481 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 13:30:05.486 INFO 4486 --- [alarm-detect-18,197e3476c6f0d052,d3807d54f8d416c1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:05.489 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.490 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.491 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.491 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 13:30:05.491 INFO 4486 --- [alarm-detect-2,197e3476c6f0d052,8af631e822b9fbe5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:05.491 INFO 4486 --- [alarm-detect-1,197e3476c6f0d052,9976bfb6c7418b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:05.497 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.497 INFO 4486 --- [alarm-detect-2,197e3476c6f0d052,8af631e822b9fbe5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:05.499 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.500 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.500 INFO 4486 --- [alarm-detect-1,197e3476c6f0d052,9976bfb6c7418b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:05.500 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 13:30:05.505 INFO 4486 --- [alarm-detect-2,197e3476c6f0d052,8af631e822b9fbe5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:05.506 INFO 4486 --- [alarm-detect-1,197e3476c6f0d052,9976bfb6c7418b38] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:05.509 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.510 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.512 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.514 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 13:30:05.524 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.525 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.529 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.530 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 13:30:05.540 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.541 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.542 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.543 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 13:30:05.549 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.550 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.551 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.552 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 13:30:05.557 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.558 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.559 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.560 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 13:30:05.565 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.566 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.567 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.568 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 13:30:05.573 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.575 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.578 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.580 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 13:30:05.594 INFO 4486 --- [alarm-detect-6,197e3476c6f0d052,e3be884d928b8d6a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:05.600 INFO 4486 --- [alarm-detect-6,197e3476c6f0d052,e3be884d928b8d6a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:05.603 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.604 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.605 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.606 INFO 4486 --- [alarm-detect-6,197e3476c6f0d052,e3be884d928b8d6a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:05.606 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 13:30:05.614 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.615 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.616 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.617 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 13:30:05.624 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.625 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.626 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.627 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 13:30:05.633 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.634 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.634 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.635 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 13:30:05.640 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.641 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.642 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.642 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 13:30:05.647 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.648 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.649 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.649 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 13:30:05.654 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.654 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.655 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.655 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 13:30:05.660 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.660 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.661 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.661 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 13:30:05.674 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.674 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.675 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.676 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 13:30:05.697 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.698 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.699 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.700 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 13:30:05.706 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.707 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.708 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.709 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 13:30:05.716 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.717 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.718 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.718 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 13:30:05.729 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.729 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.730 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.731 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 13:30:05.735 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.735 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.736 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.737 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 13:30:05.742 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.742 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.743 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.744 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 13:30:05.748 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 13:30:05.748 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 13:30:05.756 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.756 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:30:05.758 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.758 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.759 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.759 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 13:30:05.767 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.768 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:30:05.769 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:05.769 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 13:30:05.775 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 13:30:05.784 INFO 4486 --- [subtask-executor32,2233273729644a0c,13af711e63821eeb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7932261, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 13:30:06.676 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:06.716 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e550c29bae66b5b7,60e898b7fdb97267] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=e550c29bae66b5b7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:30:06.717 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e550c29bae66b5b7,60e898b7fdb97267] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=e550c29bae66b5b7, duration=2 ms] 2024-10-16 13:30:06.903 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,65e20e84a9c56192,28688ca958f74f92] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=65e20e84a9c56192, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:06.904 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,65e20e84a9c56192,28688ca958f74f92] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=65e20e84a9c56192, duration=1 ms] 2024-10-16 13:30:07.003 INFO 4486 --- [ocp-async-9,a3dffccdcd3c309c,01532269ec4662b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:07.066 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:30:07.067 INFO 4486 --- [ocp-schedules-19,29ba94b540351541,6cd569c881105490] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:30:07.068 INFO 4486 --- [ocp-schedules-19,29ba94b540351541,6cd569c881105490] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:07.081 INFO 4486 --- [ocp-schedules-17,be1945d6462cd2fa,20f181a2ba1196bd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:30:07.083 INFO 4486 --- [ocp-schedules-17,be1945d6462cd2fa,20f181a2ba1196bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:30:07.089 INFO 4486 --- [ocp-schedules-30,a84eaddfd7f1ac5a,bbda240f6ae31bb6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:30:07.089 INFO 4486 --- [ocp-schedules-25,52fd855ed31963a9,7df18272bf1b55fc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:30:07.090 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:30:07.090 INFO 4486 --- [host-check-agent-status1,52fd855ed31963a9,69255054224cdc29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:30:07.090 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:30:07.090 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:30:07.090 INFO 4486 --- [host-check-agent-status2,52fd855ed31963a9,25c9208de5625d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:30:07.091 INFO 4486 --- [host-check-agent-status4,52fd855ed31963a9,1f6be644d48f8b33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:30:07.091 INFO 4486 --- [host-check-agent-status3,52fd855ed31963a9,8a2d512bf3eea787] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:30:07.091 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:30:07.092 INFO 4486 --- [ocp-schedules-24,24a7a3de5decff33,7b08cf3661cb2213] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:30:07.093 INFO 4486 --- [host-check-clockdiff1,24a7a3de5decff33,5c40b5612bedad59] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:30:07.093 INFO 4486 --- [host-check-clockdiff2,24a7a3de5decff33,2f6b07d44c2a1dec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:30:07.093 INFO 4486 --- [host-check-clockdiff3,24a7a3de5decff33,293a101c91982416] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:30:07.095 INFO 4486 --- [host-check-clockdiff4,24a7a3de5decff33,aae20b69e49a2b01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:30:07.096 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:07.096 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:30:07.097 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:30:07.097 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:07.098 INFO 4486 --- [host-check-agent-status4,52fd855ed31963a9,1f6be644d48f8b33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:30:07.098 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:30:07.099 INFO 4486 --- [host-check-agent-status3,52fd855ed31963a9,8a2d512bf3eea787] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:30:07.099 INFO 4486 --- [host-check-agent-status1,52fd855ed31963a9,69255054224cdc29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:07.117 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:30:07.147 INFO 4486 --- [host-check-clockdiff3,24a7a3de5decff33,293a101c91982416] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056607 3 3 , error= 2024-10-16 13:30:07.147 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:30:07.149 INFO 4486 --- [host-check-clockdiff1,24a7a3de5decff33,5c40b5612bedad59] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056607 0 0 , error= 2024-10-16 13:30:07.150 INFO 4486 --- [host-check-agent-status1,52fd855ed31963a9,69255054224cdc29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:07.156 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:30:07.156 INFO 4486 --- [host-check-agent-status3,52fd855ed31963a9,8a2d512bf3eea787] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:30:07.159 INFO 4486 --- [host-check-agent-status4,52fd855ed31963a9,1f6be644d48f8b33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:30:07.161 INFO 4486 --- [host-check-agent-status2,52fd855ed31963a9,25c9208de5625d1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:30:07.163 INFO 4486 --- [host-check-agent-status2,52fd855ed31963a9,25c9208de5625d1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:30:07.166 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.167 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:07.168 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:30:07.183 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:30:07.184 INFO 4486 --- [host-check-agent-status3,52fd855ed31963a9,8a2d512bf3eea787] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:30:07.184 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:30:07.184 INFO 4486 --- [host-check-agent-status1,52fd855ed31963a9,69255054224cdc29] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:30:07.185 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:30:07.189 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:30:07.192 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:30:07.191779306+08:00 2024-10-16 13:30:07.192 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:30:07.192297410+08:00 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:30:07.192+08:00 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:30:07.195616480+08:00 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-agent-status3,52fd855ed31963a9,8a2d512bf3eea787] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-agent-status2,52fd855ed31963a9,25c9208de5625d1e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-agent-status4,52fd855ed31963a9,1f6be644d48f8b33] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-agent-status3,52fd855ed31963a9,8a2d512bf3eea787] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-agent-status3,52fd855ed31963a9,8a2d512bf3eea787] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:30:07.192+08:00 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:30:07.193+08:00 2024-10-16 13:30:07.193 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:30:07.196417224+08:00 2024-10-16 13:30:07.194 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.194 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:30:07.193+08:00 2024-10-16 13:30:07.195 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:07.196 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:30:07.192+08:00 and status AVAILABLE 2024-10-16 13:30:07.196 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:30:07.192+08:00 and status AVAILABLE 2024-10-16 13:30:07.196 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:30:07.193+08:00 and status AVAILABLE 2024-10-16 13:30:07.197 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:30:07.193+08:00 and status AVAILABLE 2024-10-16 13:30:07.197 INFO 4486 --- [host-check-agent-status4,52fd855ed31963a9,1f6be644d48f8b33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:30:07.197 INFO 4486 --- [host-check-agent-status4,52fd855ed31963a9,1f6be644d48f8b33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:30:07.197 INFO 4486 --- [host-check-agent-status4,52fd855ed31963a9,1f6be644d48f8b33] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:30:07.197 INFO 4486 --- [host-check-agent-status2,52fd855ed31963a9,25c9208de5625d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:30:07.197 INFO 4486 --- [host-check-agent-status2,52fd855ed31963a9,25c9208de5625d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:30:07.197 INFO 4486 --- [host-check-agent-status2,52fd855ed31963a9,25c9208de5625d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:30:07.198 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:30:07.198 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:30:07.198 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:30:07.199 INFO 4486 --- [host-check-agent-status1,52fd855ed31963a9,69255054224cdc29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:30:07.199 INFO 4486 --- [host-check-agent-status1,52fd855ed31963a9,69255054224cdc29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:30:07.199 INFO 4486 --- [host-check-agent-status1,52fd855ed31963a9,69255054224cdc29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:30:07.199 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:30:07.200 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:30:07.200 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.204 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:30:07.209 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:07.209 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:30:07.209 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:30:07.211 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:30:07.211 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:07.211 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:30:07.211 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:30:07.213 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:30:07.213 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.214 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:07.229 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.231 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:07.234 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:07.236 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:07.237 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:07.238 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:07.244 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.245 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:07.248 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.250 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:30:07.261 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:07.263 INFO 4486 --- [ocp-schedules-6,16036df2d03f8bb6,438d67041919d7ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:30:07.265 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:30:07.266 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:30:07.267 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:30:07.277 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:30:07.278 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:30:07.280 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:30:07.293 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:30:07.295 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:07.295 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:07.298 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:07.304 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:30:07.323 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:07.325 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:30:07.328 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:30:07.337 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:30:07.339 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:30:07.345 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:30:07.355 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:07.356 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:30:07.360 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:07.374 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:07.385 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:30:07.386 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:30:07.391 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:30:07.398 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:30:07.398 INFO 4486 --- [host-check-host-status3,a84eaddfd7f1ac5a,9bbff42e0d0d7e10] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:30:07.402 INFO 4486 --- [host-check-host-status4,a84eaddfd7f1ac5a,299c058f84c7ef5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:30:07.412 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:30:07.415 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:07.422 INFO 4486 --- [host-check-host-status2,a84eaddfd7f1ac5a,fecfc0864496dac6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:30:07.468 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:30:07.480 INFO 4486 --- [host-check-host-status1,a84eaddfd7f1ac5a,92c10f9ddf6d0f6a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:30:07.994 INFO 4486 --- [prometheus-minute-schedule-1,7cf9e8f594375ce3,4c98fe302e92fa91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:07.999 INFO 4486 --- [prometheus-minute-schedule-1,7cf9e8f594375ce3,4c98fe302e92fa91] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:08.057 INFO 4486 --- [host-check-clockdiff4,24a7a3de5decff33,aae20b69e49a2b01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056608 2 2 , error= 2024-10-16 13:30:08.060 INFO 4486 --- [host-check-clockdiff2,24a7a3de5decff33,2f6b07d44c2a1dec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056608 3 3 , error= 2024-10-16 13:30:09.004 INFO 4486 --- [ocp-async-1,da4b95a6738be438,65823f47a6190450] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:09.112 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,219e3fe21b03aefb,63542c1501bceedf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=219e3fe21b03aefb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:09.113 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,219e3fe21b03aefb,63542c1501bceedf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=219e3fe21b03aefb, duration=2 ms] 2024-10-16 13:30:09.677 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:10.006 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:30:10.006 INFO 4486 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:30:10.026 INFO 4486 --- [Check obproxy1,a2d00cc471ad001c,995c4530b1e27c6d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:30:10.026 INFO 4486 --- [Check obproxy2,a629e675f9f22c93,c534fb203e0b1dfb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:30:10.026 INFO 4486 --- [Check obproxy2,a629e675f9f22c93,c534fb203e0b1dfb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:10.026 INFO 4486 --- [Check obproxy1,a2d00cc471ad001c,995c4530b1e27c6d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:10.030 INFO 4486 --- [Collect obproxy2,9866668425557f25,3c204b4952465113] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:30:10.030 INFO 4486 --- [Collect obproxy2,9866668425557f25,3c204b4952465113] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:10.030 INFO 4486 --- [Collect obproxy1,9a40355d363f6619,c577a76857a7f27d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:30:10.030 INFO 4486 --- [Collect obproxy1,9a40355d363f6619,c577a76857a7f27d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:10.051 INFO 4486 --- [Check obproxy2,a629e675f9f22c93,c534fb203e0b1dfb] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:30:10.051 INFO 4486 --- [Check obproxy2,a629e675f9f22c93,c534fb203e0b1dfb] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:30:10.051 INFO 4486 --- [Collect obproxy2,9866668425557f25,3c204b4952465113] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:30:10.058 INFO 4486 --- [Collect obproxy2,9866668425557f25,3c204b4952465113] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:30:10.059 INFO 4486 --- [Collect obproxy1,9a40355d363f6619,c577a76857a7f27d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:30:10.059 INFO 4486 --- [Check obproxy1,a2d00cc471ad001c,995c4530b1e27c6d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:30:10.059 INFO 4486 --- [Check obproxy1,a2d00cc471ad001c,995c4530b1e27c6d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:30:10.062 INFO 4486 --- [Collect obproxy1,9a40355d363f6619,c577a76857a7f27d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:30:10.063 INFO 4486 --- [Collect obproxy2,9866668425557f25,3c204b4952465113] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:30:10.065 INFO 4486 --- [Collect obproxy1,9a40355d363f6619,c577a76857a7f27d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:30:10.180 INFO 4486 --- [ocp-schedules-30,bdb6a4fd0a2fcec6,147275608018010b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:30:10.737 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6073d04bf5ac67f1,b8808d0974f5265a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6073d04bf5ac67f1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:10.737 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6073d04bf5ac67f1,b8808d0974f5265a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6073d04bf5ac67f1, duration=1 ms] 2024-10-16 13:30:10.761 INFO 4486 --- [ocp-schedules-29,335649e394c4c3ee,708aa817192acafe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 13:30:10.761 INFO 4486 --- [ocp-schedules-29,335649e394c4c3ee,708aa817192acafe] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 13:30:10.805 INFO 4486 --- [pool-277-thread-1,335649e394c4c3ee,cf3c217a3a194af2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:30:10.805 INFO 4486 --- [pool-277-thread-3,335649e394c4c3ee,986c8a4052ca4223] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:30:10.805 INFO 4486 --- [pool-277-thread-2,335649e394c4c3ee,2588fe5e41832d50] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:30:10.825 INFO 4486 --- [pool-277-thread-1,335649e394c4c3ee,cf3c217a3a194af2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:10.829 INFO 4486 --- [pool-277-thread-1,335649e394c4c3ee,cf3c217a3a194af2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056610000000] 2024-10-16 13:30:10.830 INFO 4486 --- [pool-277-thread-2,335649e394c4c3ee,2588fe5e41832d50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:10.834 INFO 4486 --- [pool-277-thread-2,335649e394c4c3ee,2588fe5e41832d50] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056610000000] 2024-10-16 13:30:10.839 INFO 4486 --- [pool-277-thread-3,335649e394c4c3ee,986c8a4052ca4223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:10.843 INFO 4486 --- [pool-277-thread-3,335649e394c4c3ee,986c8a4052ca4223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056610000000] 2024-10-16 13:30:10.977 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:30:10.994 INFO 4486 --- [prometheus-minute-schedule-1,1884ab15d393aedc,e59c787e4bfbb5f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:10.999 INFO 4486 --- [prometheus-minute-schedule-1,1884ab15d393aedc,e59c787e4bfbb5f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:30:11.008 INFO 4486 --- [ocp-async-8,8bdc152b8ed8ac67,d0e5c7e4f27a16d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:11.118 WARN 4486 --- [pool-277-thread-3,335649e394c4c3ee,986c8a4052ca4223] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 13:30:11.118 WARN 4486 --- [pool-277-thread-3,335649e394c4c3ee,986c8a4052ca4223] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 13:30:11.120 INFO 4486 --- [pool-277-thread-3,335649e394c4c3ee,986c8a4052ca4223] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:30:11.130 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8a5b0d0531d6ba51,58a105faa331a6e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8a5b0d0531d6ba51, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:11.130 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8a5b0d0531d6ba51,58a105faa331a6e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8a5b0d0531d6ba51, duration=1 ms] 2024-10-16 13:30:11.419 WARN 4486 --- [pool-277-thread-1,335649e394c4c3ee,cf3c217a3a194af2] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 13:30:11.421 INFO 4486 --- [pool-277-thread-1,335649e394c4c3ee,cf3c217a3a194af2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:30:11.635 WARN 4486 --- [pool-277-thread-2,335649e394c4c3ee,2588fe5e41832d50] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 13:30:11.637 INFO 4486 --- [pool-277-thread-2,335649e394c4c3ee,2588fe5e41832d50] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:30:11.638 INFO 4486 --- [ocp-schedules-29,335649e394c4c3ee,708aa817192acafe] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 13:30:12.678 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:12.982 INFO 4486 --- [ocp-async-3,acc3ad2a351ee881,8aae6eeda0ff91fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:30:12.993 ERROR 4486 --- [metric-archive-second-17,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620709) Duplicate entry '669741699544190976-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.991408] [YB420AC531B8-00060B7DB46FA0C7-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620709) Duplicate entry '669741699544190976-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.991408] [YB420AC531B8-00060B7DB46FA0C7-0-0] 2024-10-16 13:30:12.995 ERROR 4486 --- [metric-archive-second-20,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=70525) Duplicate entry '586679390194700288-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994410] [YB420AC531B8-00060B7DB4195619-0-0]; nested exception is java.sql.BatchUpdateException: (conn=70525) Duplicate entry '586679390194700288-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994410] [YB420AC531B8-00060B7DB4195619-0-0] 2024-10-16 13:30:12.995 ERROR 4486 --- [metric-archive-second-10,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '669741696482349056-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994702] [YB420AC531B8-00060B7DA1ABD563-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '669741696482349056-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994702] [YB420AC531B8-00060B7DA1ABD563-0-0] 2024-10-16 13:30:12.995 ERROR 4486 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990814) Duplicate entry '586774769053536256-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994927] [YB420AC531B8-00060B7DB03B21C8-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990814) Duplicate entry '586774769053536256-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994927] [YB420AC531B8-00060B7DB03B21C8-0-0] 2024-10-16 13:30:12.995 ERROR 4486 --- [metric-archive-second-24,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1694538) Duplicate entry '586774769288417280-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994865] [YB420AC531B8-00060B7D84EAABCF-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1694538) Duplicate entry '586774769288417280-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.994865] [YB420AC531B8-00060B7D84EAABCF-0-0] 2024-10-16 13:30:12.996 ERROR 4486 --- [metric-archive-second-18,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1416399) Duplicate entry '587073990944165888-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.995545] [YB420AC531B8-00060B7DB7688395-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1416399) Duplicate entry '587073990944165888-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.995545] [YB420AC531B8-00060B7DB7688395-0-0] 2024-10-16 13:30:12.998 ERROR 4486 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=4117402) Duplicate entry '669741696503320576-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.996631] [YB420AC531B8-00060B7DBF586ACC-0-0]; nested exception is java.sql.BatchUpdateException: (conn=4117402) Duplicate entry '669741696503320576-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.996631] [YB420AC531B8-00060B7DBF586ACC-0-0] 2024-10-16 13:30:12.999 ERROR 4486 --- [metric-archive-second-26,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620710) Duplicate entry '587073990843502592-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.999095] [YB420AC531B8-00060B7DAD2B69F8-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620710) Duplicate entry '587073990843502592-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:12.999095] [YB420AC531B8-00060B7DAD2B69F8-0-0] 2024-10-16 13:30:13.000 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=243, affectRows=243 2024-10-16 13:30:13.000 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=39, affectRows=39 2024-10-16 13:30:13.001 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=39, validCount=39 2024-10-16 13:30:13.002 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=243, validCount=243 2024-10-16 13:30:13.003 ERROR 4486 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2457395) Duplicate entry '669741696473960448-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:13.002499] [YB420AC531B8-00060B7DA8188371-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2457395) Duplicate entry '669741696473960448-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:13.002499] [YB420AC531B8-00060B7DA8188371-0-0] 2024-10-16 13:30:13.003 ERROR 4486 --- [metric-archive-second-29,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620709) Duplicate entry '586679390278586368-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:13.002643] [YB420AC531B8-00060B7DB419561A-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620709) Duplicate entry '586679390278586368-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:13.002643] [YB420AC531B8-00060B7DB419561A-0-0] 2024-10-16 13:30:13.008 INFO 4486 --- [ocp-async-4,f05664fb37594fd0,3a146dc5a14ddc9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:13.009 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=317, affectRows=317 2024-10-16 13:30:13.012 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=315, validCount=315 2024-10-16 13:30:13.018 ERROR 4486 --- [metric-archive-second-30,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '587073990931582977-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:13.017784] [YB420AC531B8-00060B7DBF586ACD-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '587073990931582977-1729056540' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:30:13.017784] [YB420AC531B8-00060B7DBF586ACD-0-0] 2024-10-16 13:30:13.105 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=153, affectRows=153 2024-10-16 13:30:13.106 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=153, validCount=153 2024-10-16 13:30:13.139 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=197, affectRows=197 2024-10-16 13:30:13.141 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=197, validCount=197 2024-10-16 13:30:13.160 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=222, affectRows=222 2024-10-16 13:30:13.161 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=222, validCount=222 2024-10-16 13:30:13.163 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=232, affectRows=232 2024-10-16 13:30:13.164 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=232, validCount=232 2024-10-16 13:30:13.170 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=248, affectRows=248 2024-10-16 13:30:13.171 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=248, validCount=248 2024-10-16 13:30:13.175 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=237, affectRows=237 2024-10-16 13:30:13.176 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=237, validCount=237 2024-10-16 13:30:13.178 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=255, affectRows=255 2024-10-16 13:30:13.179 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=255, validCount=255 2024-10-16 13:30:13.189 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=263, affectRows=263 2024-10-16 13:30:13.189 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=287, affectRows=287 2024-10-16 13:30:13.191 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=263, validCount=263 2024-10-16 13:30:13.191 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=287, validCount=287 2024-10-16 13:30:13.200 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=283, affectRows=283 2024-10-16 13:30:13.202 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=282, validCount=282 2024-10-16 13:30:13.279 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056540, nodeCount=475, affectRows=475 2024-10-16 13:30:13.281 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056540, nodeCount=475, validCount=475 2024-10-16 13:30:13.994 INFO 4486 --- [prometheus-minute-schedule-1,f80efcb9618c433a,75ef4f4be467d857] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:14.003 INFO 4486 --- [prometheus-minute-schedule-1,f80efcb9618c433a,75ef4f4be467d857] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:14.303 INFO 4486 --- [metric-archive-async-24,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056540, nodeCount=45, affectRows=45 2024-10-16 13:30:14.307 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056540, nodeCount=143, affectRows=143 2024-10-16 13:30:14.308 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056540, nodeCount=125, affectRows=125 2024-10-16 13:30:14.308 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056540, nodeCount=164, affectRows=164 2024-10-16 13:30:14.381 INFO 4486 --- [metric-archive-async-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056540, nodeCount=45, validCount=45 2024-10-16 13:30:14.507 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056540, nodeCount=125, validCount=125 2024-10-16 13:30:14.523 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056540, nodeCount=143, validCount=143 2024-10-16 13:30:14.554 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056540, nodeCount=164, validCount=164 2024-10-16 13:30:15.008 INFO 4486 --- [ocp-async-1,e88b05b2bf10ab42,9c5cc7270c988325] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:15.626 INFO 4486 --- [ocp-schedules-29,0d6facf81843b066,521a94a30750977b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:30:15.681 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:15.705 WARN 4486 --- [scanner-13241519-1,0d6facf81843b066,b259e39546bac743] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:30:15.732 INFO 4486 --- [alarm-detect-15,0d6facf81843b066,88ee34c474fd2ec3] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056585 ~ 1729056605 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056600, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056600, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056600=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:30:16.908 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cbe4afcb2cc7d1d2,21e558f2cbdae61b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cbe4afcb2cc7d1d2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:16.909 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cbe4afcb2cc7d1d2,21e558f2cbdae61b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cbe4afcb2cc7d1d2, duration=3 ms] 2024-10-16 13:30:16.994 INFO 4486 --- [prometheus-minute-schedule-1,7b9b75cef8ec3287,679512e4db646416] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:17.000 INFO 4486 --- [prometheus-minute-schedule-1,7b9b75cef8ec3287,679512e4db646416] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:17.009 INFO 4486 --- [ocp-async-7,cfce6d171551432a,37a9b7140dca9c14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:17.062 INFO 4486 --- [ocp-schedules-1,dc882ac5c67e5c77,6ba272067c0b2c9a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:30:17.070 INFO 4486 --- [ocp-schedules-19,29ba94b540351541,6cd569c881105490] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:30:17.200 INFO 4486 --- [ocp-schedules-17,be1945d6462cd2fa,20f181a2ba1196bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:30:18.682 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:18.809 INFO 4486 --- [ocp-schedules-27,df908e2b086b9fd5,3f80d8c6e4ee186e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:30:19.009 INFO 4486 --- [ocp-async-6,8f7147fead889b7a,bf5cb8389f72c33d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:19.117 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,702b02975e80d2e4,c89e62a93598c557] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=702b02975e80d2e4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:19.117 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,702b02975e80d2e4,c89e62a93598c557] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=702b02975e80d2e4, duration=1 ms] 2024-10-16 13:30:19.994 INFO 4486 --- [prometheus-minute-schedule-1,91e0a4500854ef74,aaae8faa4006d6f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:20.000 INFO 4486 --- [prometheus-minute-schedule-1,91e0a4500854ef74,aaae8faa4006d6f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:20.185 INFO 4486 --- [ocp-schedules-24,38ca089f5f488ddb,e62a92f2a4ca0051] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:30:20.743 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,14c99cd8df3b5513,af9cfd75f87ed9c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=14c99cd8df3b5513, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:20.744 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,14c99cd8df3b5513,af9cfd75f87ed9c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=14c99cd8df3b5513, duration=1 ms] 2024-10-16 13:30:20.810 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:30:20.996 INFO 4486 --- [prometheus-cache-monitor1,e7cb1380ac3099b5,41529be8fa9286a0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13855,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:30:20.996 INFO 4486 --- [prometheus-cache-monitor1,e7cb1380ac3099b5,41529be8fa9286a0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:30:21.009 INFO 4486 --- [ocp-async-4,106bcbe52db73a2a,b204119eaf47a7bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:21.020 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,43a9bd14092d52be,3b07baaf6cf66156] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=43a9bd14092d52be, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:30:21.021 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,43a9bd14092d52be,3b07baaf6cf66156] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=43a9bd14092d52be, duration=2 ms] 2024-10-16 13:30:21.134 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1fafd5a2bbc15498,3b73e895a9afcf6c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1fafd5a2bbc15498, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:21.134 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1fafd5a2bbc15498,3b73e895a9afcf6c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1fafd5a2bbc15498, duration=1 ms] 2024-10-16 13:30:21.683 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:22.628 INFO 4486 --- [ocp-schedules-9,313d6d00c58c7d4e,4816ead39b537529] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:30:22.982 INFO 4486 --- [ocp-async-1,f1ff648a8c30bb01,649d88505939ad72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:30:22.994 INFO 4486 --- [prometheus-minute-schedule-1,12cde133c807d23a,7233a84549d71c3b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:22.999 INFO 4486 --- [prometheus-minute-schedule-1,12cde133c807d23a,7233a84549d71c3b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:23.009 INFO 4486 --- [ocp-async-2,a4a98fbfd9b6ea39,02176d6f63925155] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:23.396 INFO 4486 --- [ocp-schedules-26,ed14f533f1405471,eb00bb7a60e034a8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:30:23.398 INFO 4486 --- [ocp-schedules-26,ed14f533f1405471,eb00bb7a60e034a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:23.433 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:30:23.433 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.438 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:30:23.447 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.447 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:30:23.452 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:30:23.453 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.457 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.457 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:30:23.457 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:30:23.465 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.465 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:30:23.476 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.476 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.476 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:30:23.477 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:30:23.484 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.485 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:23.487 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.488 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:30:23.489 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:30:23.489 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.494 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:30:23.500 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.500 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:23.505 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.506 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.506 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:23.507 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:30:23.516 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.517 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:30:23.519 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.520 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:23.522 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.523 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:30:23.525 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.526 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:30:23.533 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.534 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:30:23.538 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.539 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:30:23.542 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.543 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:30:23.545 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.546 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:30:23.554 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.555 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:30:23.555 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.556 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.557 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.557 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.558 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.559 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.559 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.560 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.561 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.562 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.562 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.563 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.564 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.564 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.564 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.565 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:30:23.565 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.566 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.566 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.567 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.567 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.568 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:23.568 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.569 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.569 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.570 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.570 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.571 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.571 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.571 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.572 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:30:23.573 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.574 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.577 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.579 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:23.579 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:30:23.581 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.582 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.584 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.585 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.588 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.589 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.590 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.591 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:23.592 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.593 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.596 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.597 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.599 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:23.599 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.599 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.600 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:30:23.601 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.608 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.609 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.610 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:30:23.611 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:30:23.613 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:30:23.614 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:30:23.614 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:30:23.622 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.623 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.626 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.627 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.630 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.631 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.634 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.635 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.638 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.639 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.641 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.642 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:23.642 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.644 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.645 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:30:23.645 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.646 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:30:23.649 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.650 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:30:23.653 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.654 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:30:23.657 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:30:23.661 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:30:23.665 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.665 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:30:23.665 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.665 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:30:23.665 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.665 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:30:23.670 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:30:23.670 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:30:23.720 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:23.721 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:23.722 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:30:23.732 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.732 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:30:23.732 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.732 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:30:23.732 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:30:23.732 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:30:23.736 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:30:23.736 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:30:24.071 INFO 4486 --- [ocp-schedules-28,11e1c987d66de8e6,be623d6a105e0b1c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:30:24.232 INFO 4486 --- [ocp-schedules-18,fb3829e6fb456af8,3c148aca3be02bbb] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:30:24.232 INFO 4486 --- [ocp-schedules-18,fb3829e6fb456af8,3c148aca3be02bbb] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:30:24.265 INFO 4486 --- [Collect all tenants major compactions2,fb3829e6fb456af8,284a9220a875c561] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:30:24.265 INFO 4486 --- [Collect all tenants major compactions3,fb3829e6fb456af8,d742ad492fc648f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:30:24.265 INFO 4486 --- [Collect all tenants major compactions4,fb3829e6fb456af8,881a1009c5ba4d7c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:30:24.266 INFO 4486 --- [Collect all tenants major compactions5,fb3829e6fb456af8,0bcee511a9e46499] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:30:24.267 INFO 4486 --- [Collect all tenants major compactions1,fb3829e6fb456af8,957a13ee17b44258] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:30:24.277 INFO 4486 --- [Collect all tenants major compactions2,fb3829e6fb456af8,284a9220a875c561] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:24.278 INFO 4486 --- [Collect all tenants major compactions2,fb3829e6fb456af8,284a9220a875c561] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:30:24.280 INFO 4486 --- [Collect all tenants major compactions1,fb3829e6fb456af8,957a13ee17b44258] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:24.282 INFO 4486 --- [Collect all tenants major compactions3,fb3829e6fb456af8,d742ad492fc648f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:24.282 INFO 4486 --- [Collect all tenants major compactions1,fb3829e6fb456af8,957a13ee17b44258] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:30:24.282 INFO 4486 --- [Collect all tenants major compactions2,fb3829e6fb456af8,284a9220a875c561] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:30:24.283 INFO 4486 --- [Collect all tenants major compactions2,fb3829e6fb456af8,284a9220a875c561] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:30:24.285 INFO 4486 --- [Collect all tenants major compactions3,fb3829e6fb456af8,d742ad492fc648f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:30:24.285 INFO 4486 --- [Collect all tenants major compactions1,fb3829e6fb456af8,957a13ee17b44258] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:30:24.285 INFO 4486 --- [Collect all tenants major compactions1,fb3829e6fb456af8,957a13ee17b44258] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:30:24.289 INFO 4486 --- [Collect all tenants major compactions3,fb3829e6fb456af8,d742ad492fc648f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:30:24.289 INFO 4486 --- [Collect all tenants major compactions3,fb3829e6fb456af8,d742ad492fc648f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:30:24.296 INFO 4486 --- [Collect all tenants major compactions4,fb3829e6fb456af8,881a1009c5ba4d7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:24.296 INFO 4486 --- [Collect all tenants major compactions5,fb3829e6fb456af8,0bcee511a9e46499] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:24.297 INFO 4486 --- [Collect all tenants major compactions4,fb3829e6fb456af8,881a1009c5ba4d7c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:30:24.297 INFO 4486 --- [Collect all tenants major compactions5,fb3829e6fb456af8,0bcee511a9e46499] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:30:24.301 INFO 4486 --- [Collect all tenants major compactions4,fb3829e6fb456af8,881a1009c5ba4d7c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:30:24.301 INFO 4486 --- [Collect all tenants major compactions4,fb3829e6fb456af8,881a1009c5ba4d7c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:30:24.302 INFO 4486 --- [Collect all tenants major compactions5,fb3829e6fb456af8,0bcee511a9e46499] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:30:24.302 INFO 4486 --- [Collect all tenants major compactions5,fb3829e6fb456af8,0bcee511a9e46499] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:30:24.303 INFO 4486 --- [ocp-schedules-18,fb3829e6fb456af8,3c148aca3be02bbb] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:30:24.684 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:25.009 INFO 4486 --- [ocp-async-3,97b432753638ce6b,11484d975a2d1212] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:25.793 INFO 4486 --- [ocp-schedules-18,38d9e80dd334f9c5,974825304a55fd1e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:30:25.866 WARN 4486 --- [scanner-13241752-1,38d9e80dd334f9c5,50611b9b0b4756d6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:30:25.897 INFO 4486 --- [alarm-detect-14,38d9e80dd334f9c5,0c6fca76b828cb25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:25.903 INFO 4486 --- [alarm-detect-14,38d9e80dd334f9c5,0c6fca76b828cb25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:25.905 INFO 4486 --- [alarm-detect-10,38d9e80dd334f9c5,ed20836a9735203c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:25.906 INFO 4486 --- [alarm-detect-3,38d9e80dd334f9c5,dc40daed72b2f1b7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:25.908 INFO 4486 --- [alarm-detect-14,38d9e80dd334f9c5,0c6fca76b828cb25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:25.910 INFO 4486 --- [alarm-detect-3,38d9e80dd334f9c5,dc40daed72b2f1b7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:25.911 INFO 4486 --- [alarm-detect-10,38d9e80dd334f9c5,ed20836a9735203c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:25.915 INFO 4486 --- [alarm-detect-10,38d9e80dd334f9c5,ed20836a9735203c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:25.915 INFO 4486 --- [alarm-detect-3,38d9e80dd334f9c5,dc40daed72b2f1b7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:25.927 INFO 4486 --- [alarm-detect-18,38d9e80dd334f9c5,3e925c00faa42552] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056600 ~ 1729056615 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056615, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056615, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056615=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:30:25.981 INFO 4486 --- [alarm-detect-18,38d9e80dd334f9c5,26dc5adc2ec47296] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:25.985 INFO 4486 --- [alarm-detect-18,38d9e80dd334f9c5,26dc5adc2ec47296] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:25.988 INFO 4486 --- [alarm-detect-18,38d9e80dd334f9c5,26dc5adc2ec47296] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:25.994 INFO 4486 --- [prometheus-minute-schedule-1,dc0d16662f52ea24,7ddc936446c06114] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:26.002 INFO 4486 --- [prometheus-minute-schedule-1,dc0d16662f52ea24,7ddc936446c06114] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:26.214 INFO 4486 --- [ocp-schedules-16,b18c017d9cb2654c,001958d98e52459a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:30:26.214 INFO 4486 --- [ocp-schedules-16,aed55b2b48b70833,a71093d831466719] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:30:26.216 INFO 4486 --- [ocp-schedules-16,aed55b2b48b70833,a71093d831466719] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:26.244 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.245 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:26.253 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.254 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:26.278 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.278 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:26.283 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.284 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.284 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:30:26.319 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.319 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.320 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:26.321 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:30:26.321 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.324 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:30:26.325 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:26.335 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.335 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:26.345 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.354 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.354 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.355 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:26.356 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:30:26.360 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:30:26.364 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:26.366 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:26.387 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.389 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.390 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:30:26.393 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:30:26.414 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:30:26.414 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:30:26.439 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.440 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.441 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:26.444 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.448 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:30:26.451 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:26.467 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:30:26.478 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.480 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.480 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:30:26.482 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.482 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:26.491 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:30:26.491 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:30:26.513 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.513 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:26.540 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.540 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:26.573 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.573 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:26.607 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.607 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:26.638 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:26.667 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.669 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.670 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:26.674 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:30:26.679 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:30:26.684 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:30:26.687 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:30:26.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:30:26.747 INFO 4486 --- [ocp-schedules-3,49969ff83f3e9d2e,fd6f77c68eeeef52] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:30:26.747+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:30:26.749 INFO 4486 --- [ocp-schedules-3,49969ff83f3e9d2e,fd6f77c68eeeef52] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:30:26.749 INFO 4486 --- [ocp-schedules-3,49969ff83f3e9d2e,fd6f77c68eeeef52] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:30:26.750 INFO 4486 --- [ocp-schedules-3,49969ff83f3e9d2e,fd6f77c68eeeef52] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:30:26.833 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:30:26.860 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:26.861 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:26.862 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:30:26.892 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:30:26.892 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:30:26.915 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,705b3e0f7a9f164d,53916202d1f32bd1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=705b3e0f7a9f164d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:26.915 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,705b3e0f7a9f164d,53916202d1f32bd1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=705b3e0f7a9f164d, duration=1 ms] 2024-10-16 13:30:27.010 INFO 4486 --- [ocp-async-10,969a98976db3de9b,bd8d8115ff096d66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:27.685 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:28.994 INFO 4486 --- [prometheus-minute-schedule-1,df2831c6b285c8ef,0f2f47957a21d60c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:29.000 INFO 4486 --- [prometheus-minute-schedule-1,df2831c6b285c8ef,0f2f47957a21d60c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:29.010 INFO 4486 --- [ocp-async-8,e5ef4def985726e1,77441a27f6c49e19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:29.126 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,79757ee324547b7f,e91b0879f027703b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=79757ee324547b7f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:29.126 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,79757ee324547b7f,e91b0879f027703b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=79757ee324547b7f, duration=1 ms] 2024-10-16 13:30:30.103 INFO 4486 --- [prometheus-second-schedule-1,4b93f9416d6c378f,61e7dc9a7d16af34] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:30:30.190 INFO 4486 --- [ocp-schedules-19,07e7de9362e6d5f7,b463b4c252b1a977] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:30:30.685 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:30.750 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e423e1df03eaeb27,ba13baa785a83897] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e423e1df03eaeb27, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:30.750 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e423e1df03eaeb27,ba13baa785a83897] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e423e1df03eaeb27, duration=2 ms] 2024-10-16 13:30:31.010 INFO 4486 --- [ocp-async-9,521770a43da6c877,189a2b0190d3a53b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:31.143 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d376535aa9ed317e,10cb780ebb3bca90] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d376535aa9ed317e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:31.143 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d376535aa9ed317e,10cb780ebb3bca90] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d376535aa9ed317e, duration=2 ms] 2024-10-16 13:30:31.994 INFO 4486 --- [prometheus-minute-schedule-1,fae472ef11f6ef28,328ae54d9666ebed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:32.000 INFO 4486 --- [prometheus-minute-schedule-1,fae472ef11f6ef28,328ae54d9666ebed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:32.983 INFO 4486 --- [ocp-async-3,11b9d161c4b14cfd,c8df491f580fdebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:30:33.010 INFO 4486 --- [ocp-async-10,65b0c68e33376190,8c14563de089f92d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:33.686 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:34.994 INFO 4486 --- [prometheus-minute-schedule-1,9ff2fae7dfa19084,71f8ff3d280a2de3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:35.001 INFO 4486 --- [prometheus-minute-schedule-1,9ff2fae7dfa19084,71f8ff3d280a2de3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:35.010 INFO 4486 --- [ocp-async-4,11370ea5b3f5f584,ead31010d2e56836] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:36.017 INFO 4486 --- [ocp-schedules-30,3c315aafe0eacdf8,4a895931023db33f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:30:36.054 WARN 4486 --- [scanner-13242104-1,3c315aafe0eacdf8,8a0c45dd17f90f18] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:30:36.687 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:36.920 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8b63ec8600d2262b,24fc4887443d257c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8b63ec8600d2262b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:36.920 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8b63ec8600d2262b,24fc4887443d257c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8b63ec8600d2262b, duration=1 ms] 2024-10-16 13:30:37.011 INFO 4486 --- [ocp-async-2,e4f4f1c535b3a92d,af5cb1769c1608a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:37.063 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:30:37.064 INFO 4486 --- [ocp-schedules-2,97dbf10ed9217b30,83f0e6b3cf4762b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:30:37.066 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:30:37.066 INFO 4486 --- [ocp-schedules-5,9ed07f3e938c6e08,e9f23401a12c8401] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:30:37.069 INFO 4486 --- [ocp-schedules-5,9ed07f3e938c6e08,e9f23401a12c8401] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:37.069 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:30:37.070 INFO 4486 --- [ocp-schedules-2,97dbf10ed9217b30,83f0e6b3cf4762b8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:30:37.081 INFO 4486 --- [ocp-schedules-24,b9abb4a505af44f5,0cd5991e2e144137] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:30:37.083 INFO 4486 --- [ocp-schedules-24,b9abb4a505af44f5,0cd5991e2e144137] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:30:37.090 INFO 4486 --- [ocp-schedules-19,1de045cbd47e822c,bb60434cfbbc695a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:30:37.090 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:30:37.090 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:30:37.091 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:30:37.091 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:30:37.091 INFO 4486 --- [ocp-schedules-12,bb949eed6c8f50ff,bf0c1b86968fb27d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:30:37.092 INFO 4486 --- [host-check-clockdiff1,bb949eed6c8f50ff,8ea0e7e4b24fe4ac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:30:37.092 INFO 4486 --- [host-check-clockdiff2,bb949eed6c8f50ff,1d4cc38e3a575ad9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:30:37.092 INFO 4486 --- [ocp-schedules-30,507a0e3d59d81335,a8e171ea06351c63] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:30:37.093 INFO 4486 --- [host-check-clockdiff3,bb949eed6c8f50ff,f33b67e3569542e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:30:37.094 INFO 4486 --- [host-check-clockdiff4,bb949eed6c8f50ff,c126bb7772798c19] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:30:37.094 INFO 4486 --- [host-check-agent-status1,507a0e3d59d81335,8b6340337450aacc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:30:37.094 INFO 4486 --- [host-check-agent-status2,507a0e3d59d81335,7377a90484907c06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:30:37.098 INFO 4486 --- [host-check-agent-status3,507a0e3d59d81335,12b8fe03b56d64e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:30:37.099 INFO 4486 --- [host-check-agent-status4,507a0e3d59d81335,615a020e5224608e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:30:37.101 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:37.101 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:30:37.128 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:30:37.138 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:37.138 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:30:37.138 INFO 4486 --- [host-check-agent-status4,507a0e3d59d81335,615a020e5224608e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:30:37.139 INFO 4486 --- [host-check-agent-status3,507a0e3d59d81335,12b8fe03b56d64e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:30:37.139 INFO 4486 --- [host-check-agent-status1,507a0e3d59d81335,8b6340337450aacc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:37.139 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:30:37.143 INFO 4486 --- [host-check-agent-status4,507a0e3d59d81335,615a020e5224608e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:30:37.143 INFO 4486 --- [host-check-agent-status2,507a0e3d59d81335,7377a90484907c06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:30:37.143 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:30:37.143 INFO 4486 --- [host-check-agent-status3,507a0e3d59d81335,12b8fe03b56d64e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:30:37.144 INFO 4486 --- [host-check-clockdiff1,bb949eed6c8f50ff,8ea0e7e4b24fe4ac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056637 0 0 , error= 2024-10-16 13:30:37.147 INFO 4486 --- [host-check-agent-status2,507a0e3d59d81335,7377a90484907c06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:30:37.147 INFO 4486 --- [host-check-clockdiff4,bb949eed6c8f50ff,c126bb7772798c19] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056637 3 3 , error= 2024-10-16 13:30:37.147 INFO 4486 --- [host-check-clockdiff3,bb949eed6c8f50ff,f33b67e3569542e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056637 3 3 , error= 2024-10-16 13:30:37.147 INFO 4486 --- [host-check-agent-status1,507a0e3d59d81335,8b6340337450aacc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:37.147 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:30:37.155 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.156 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:37.157 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:30:37.171 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:30:37.171 INFO 4486 --- [host-check-agent-status1,507a0e3d59d81335,8b6340337450aacc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:30:37.172 INFO 4486 --- [host-check-agent-status3,507a0e3d59d81335,12b8fe03b56d64e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:30:37.173 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.174 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:30:37.174 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:30:37.173479781+08:00 2024-10-16 13:30:37.174 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:37.174 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:30:37.174 INFO 4486 --- [host-check-agent-status2,507a0e3d59d81335,7377a90484907c06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:30:37.174 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:30:37.174+08:00 2024-10-16 13:30:37.176 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:30:37.179211867+08:00 2024-10-16 13:30:37.176 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:30:37.179305755+08:00 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-agent-status3,507a0e3d59d81335,12b8fe03b56d64e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-agent-status3,507a0e3d59d81335,12b8fe03b56d64e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-agent-status3,507a0e3d59d81335,12b8fe03b56d64e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-agent-status1,507a0e3d59d81335,8b6340337450aacc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-agent-status1,507a0e3d59d81335,8b6340337450aacc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-agent-status1,507a0e3d59d81335,8b6340337450aacc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:30:37.176+08:00 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:30:37.177+08:00 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:30:37.174+08:00 and status AVAILABLE 2024-10-16 13:30:37.177 INFO 4486 --- [host-check-agent-status4,507a0e3d59d81335,615a020e5224608e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:30:37.178 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:30:37.179 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:30:37.180 INFO 4486 --- [host-check-agent-status2,507a0e3d59d81335,7377a90484907c06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:30:37.180 INFO 4486 --- [host-check-agent-status2,507a0e3d59d81335,7377a90484907c06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:30:37.180 INFO 4486 --- [host-check-agent-status2,507a0e3d59d81335,7377a90484907c06] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:30:37.180 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:30:37.177+08:00 and status AVAILABLE 2024-10-16 13:30:37.180 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:30:37.176+08:00 and status AVAILABLE 2024-10-16 13:30:37.181 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:30:37.186 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.186 INFO 4486 --- [host-check-agent-status4,507a0e3d59d81335,615a020e5224608e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:30:37.186 INFO 4486 --- [host-check-agent-status4,507a0e3d59d81335,615a020e5224608e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:30:37.186 INFO 4486 --- [host-check-agent-status4,507a0e3d59d81335,615a020e5224608e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:30:37.186 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:30:37.186 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:30:37.187 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.188 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:30:37.188 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:30:37.191127819+08:00 2024-10-16 13:30:37.189 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:30:37.188+08:00 2024-10-16 13:30:37.190 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.190 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.191 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:30:37.192 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:30:37.188+08:00 and status AVAILABLE 2024-10-16 13:30:37.193 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:37.194 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:30:37.194 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:30:37.194 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:30:37.194 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:30:37.195 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:37.195 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:30:37.196 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:30:37.201 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.201 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.201 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.201 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.201 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.202 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.202 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.203 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:37.203 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:30:37.205 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:30:37.212 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.213 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:37.217 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:37.223 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:37.224 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:37.224 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:30:37.227 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.228 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:30:37.230 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.231 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:30:37.232 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:30:37.241 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.242 INFO 4486 --- [ocp-schedules-20,82eabddca446add3,934f7e212ee8c9aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:30:37.260 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:30:37.263 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:30:37.263 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:30:37.270 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:30:37.273 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:30:37.274 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:30:37.279 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:30:37.286 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:37.289 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:37.291 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:30:37.292 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:37.304 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:30:37.321 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:30:37.328 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:30:37.331 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:30:37.332 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:30:37.338 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:30:37.346 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:30:37.348 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:37.356 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:37.363 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:37.363 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:30:37.374 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:30:37.377 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:30:37.390 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:30:37.390 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:30:37.390 INFO 4486 --- [host-check-host-status3,1de045cbd47e822c,5b97c055c8a787c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:30:37.400 INFO 4486 --- [host-check-host-status4,1de045cbd47e822c,f00d0c11f38b771b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:30:37.401 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:30:37.409 INFO 4486 --- [host-check-host-status2,1de045cbd47e822c,c13b2dfdf3b2733d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:30:37.444 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:37.446 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.447 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:30:37.448 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.448 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.449 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:30:37.451 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:30:37.458 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.458 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.458 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.458 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.458 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.459 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.464 INFO 4486 --- [host-check-host-status1,1de045cbd47e822c,b45c9232782bdd2c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:30:37.487 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:30:37.526 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:30:37.578 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:37.595 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:37.596 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.598 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:30:37.599 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:30:37.600 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.601 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:30:37.602 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.603 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.604 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:30:37.612 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.612 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.612 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.612 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.612 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.613 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.620 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:30:37.814 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:30:37.860 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:37.865 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.866 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:30:37.868 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.868 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.871 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:30:37.882 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.882 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.882 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.882 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.882 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:37.883 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:37.901 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:30:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:30:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:30:37.994 INFO 4486 --- [prometheus-minute-schedule-1,8aec88770cc007c1,7168baf8979937a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:38.001 INFO 4486 --- [prometheus-minute-schedule-1,8aec88770cc007c1,7168baf8979937a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:38.072 INFO 4486 --- [host-check-clockdiff2,bb949eed6c8f50ff,1d4cc38e3a575ad9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056638 3 3 , error= 2024-10-16 13:30:38.140 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:30:38.187 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:38.190 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:38.191 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:30:38.192 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:38.193 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:38.194 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:30:38.202 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:38.202 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:38.202 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:38.202 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:38.202 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:30:38.203 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:38.266 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:30:38.470 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:30:38.269811, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:30:38.471 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:30:38.269811, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:30:38.525 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:30:38.269811} } 2024-10-16 13:30:38.591 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:30:38.598 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:30:38.601 INFO 4486 --- [ocp-schedules-17,88050889494eafe9,8f811388893be46d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:30:38.814 INFO 4486 --- [ocp-schedules-21,6a9ffd3135a5a3eb,bf4d071728c4e420] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:30:39.011 INFO 4486 --- [ocp-async-10,368318ce1e978755,b8ced25cc2c56e25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:39.131 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,310af1423c2d67a4,95dbb559d6ebe7b8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=310af1423c2d67a4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:39.131 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,310af1423c2d67a4,95dbb559d6ebe7b8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=310af1423c2d67a4, duration=1 ms] 2024-10-16 13:30:39.688 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:40.006 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:30:40.007 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:30:40.026 INFO 4486 --- [Check obproxy1,e92e8f5d2ca78639,4bd78e674fff21e9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:30:40.026 INFO 4486 --- [Check obproxy1,e92e8f5d2ca78639,4bd78e674fff21e9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:40.028 INFO 4486 --- [Check obproxy2,f5afef5eeb099079,f748953ab1134aa5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:30:40.028 INFO 4486 --- [Check obproxy2,f5afef5eeb099079,f748953ab1134aa5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:40.037 INFO 4486 --- [Collect obproxy2,2917e11a2a38b599,b429035f37e1ef2a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:30:40.037 INFO 4486 --- [Collect obproxy2,2917e11a2a38b599,b429035f37e1ef2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:40.038 INFO 4486 --- [Collect obproxy1,b97ec54a46228361,7123d2b3823156ed] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:30:40.038 INFO 4486 --- [Collect obproxy1,b97ec54a46228361,7123d2b3823156ed] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:30:40.055 INFO 4486 --- [Check obproxy1,e92e8f5d2ca78639,4bd78e674fff21e9] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:30:40.055 INFO 4486 --- [Check obproxy1,e92e8f5d2ca78639,4bd78e674fff21e9] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:30:40.057 INFO 4486 --- [Check obproxy2,f5afef5eeb099079,f748953ab1134aa5] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:30:40.057 INFO 4486 --- [Check obproxy2,f5afef5eeb099079,f748953ab1134aa5] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:30:40.062 INFO 4486 --- [Collect obproxy2,2917e11a2a38b599,b429035f37e1ef2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:30:40.066 INFO 4486 --- [Collect obproxy1,b97ec54a46228361,7123d2b3823156ed] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:30:40.067 INFO 4486 --- [Collect obproxy2,2917e11a2a38b599,b429035f37e1ef2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:30:40.069 INFO 4486 --- [Collect obproxy1,b97ec54a46228361,7123d2b3823156ed] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:30:40.072 INFO 4486 --- [Collect obproxy2,2917e11a2a38b599,b429035f37e1ef2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:30:40.072 INFO 4486 --- [Collect obproxy1,b97ec54a46228361,7123d2b3823156ed] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:30:40.196 INFO 4486 --- [ocp-schedules-21,03c07d28a9371e05,9cb654f8116f7373] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:30:40.757 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eccfe253f253fb60,33c0a669971b3405] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eccfe253f253fb60, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:40.757 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eccfe253f253fb60,33c0a669971b3405] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eccfe253f253fb60, duration=1 ms] 2024-10-16 13:30:40.994 INFO 4486 --- [prometheus-minute-schedule-1,bc70f6f68d962a2a,ebe7bc6617564b6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:40.995 INFO 4486 --- [prometheus-hour-schedule-1,c3f2eae304441ea8,eb6fd6be1018d33c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:30:41.000 INFO 4486 --- [prometheus-minute-schedule-1,bc70f6f68d962a2a,ebe7bc6617564b6f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:41.011 INFO 4486 --- [ocp-async-4,a62c64dfa3e2c874,9814f4cd84920ff2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:41.147 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3c82bbc5086b6d54,42499f7556313d7f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3c82bbc5086b6d54, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:41.147 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3c82bbc5086b6d54,42499f7556313d7f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3c82bbc5086b6d54, duration=1 ms] 2024-10-16 13:30:42.074 INFO 4486 --- [ocp-schedules-27,0d6d0a8d0c3d7521,317b00b31c440b7a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:30:42.246 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,babbaf34485908f1,8b2a9fb2c0bb8344] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=babbaf34485908f1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:30:42.247 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,babbaf34485908f1,8b2a9fb2c0bb8344] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=babbaf34485908f1, duration=2 ms] 2024-10-16 13:30:42.689 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:42.983 INFO 4486 --- [ocp-async-7,998c34b021bfa0e6,de3cc5b61a00b2e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:30:43.011 INFO 4486 --- [ocp-async-3,f72dff30a3d73ece,3ec2320ae9b53060] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:43.994 INFO 4486 --- [prometheus-minute-schedule-1,181aea3d56f00978,42a3c7d92e601afb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:44.001 INFO 4486 --- [prometheus-minute-schedule-1,181aea3d56f00978,42a3c7d92e601afb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:45.011 INFO 4486 --- [ocp-async-5,e2832ccb29dc96f1,93f7a88bae0d4c2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:45.426 INFO 4486 --- [ocp-schedules-13,88a3fed2e8f56e50,783324749f67b571] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:30:45.431 INFO 4486 --- [ocp-schedules-13,88a3fed2e8f56e50,783324749f67b571] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:30:45.431 INFO 4486 --- [ocp-schedules-13,88a3fed2e8f56e50,783324749f67b571] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:30:45.440 INFO 4486 --- [ocp-schedules-27,106d35d212d3cc24,9908ef00c693bc12] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:30:45.440 INFO 4486 --- [ocp-schedules-27,106d35d212d3cc24,9908ef00c693bc12] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:30:45.690 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:46.150 INFO 4486 --- [ocp-schedules-19,18faec680e765c5b,a4a18cce074d7b26] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:30:46.185 WARN 4486 --- [scanner-13242315-1,18faec680e765c5b,3a310d8c933b90d7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:30:46.219 INFO 4486 --- [alarm-detect-13,18faec680e765c5b,a953f6b985b260c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:46.224 INFO 4486 --- [alarm-detect-13,18faec680e765c5b,a953f6b985b260c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:46.228 INFO 4486 --- [alarm-detect-13,18faec680e765c5b,a953f6b985b260c8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:30:46.241 INFO 4486 --- [alarm-detect-17,18faec680e765c5b,4da6e02921b329d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:46.241 INFO 4486 --- [alarm-detect-6,18faec680e765c5b,b938e5248e2eadb6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:46.245 INFO 4486 --- [alarm-detect-6,18faec680e765c5b,b938e5248e2eadb6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:46.245 INFO 4486 --- [alarm-detect-17,18faec680e765c5b,4da6e02921b329d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:46.249 INFO 4486 --- [alarm-detect-17,18faec680e765c5b,4da6e02921b329d8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:30:46.249 INFO 4486 --- [alarm-detect-6,18faec680e765c5b,b938e5248e2eadb6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:30:46.286 INFO 4486 --- [alarm-detect-10,18faec680e765c5b,3bc50570fb0c125a] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056615 ~ 1729056636 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056630, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056630, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056630=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:30:46.337 INFO 4486 --- [alarm-detect-11,18faec680e765c5b,00ddda4d23b31723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:46.342 INFO 4486 --- [alarm-detect-11,18faec680e765c5b,00ddda4d23b31723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:46.347 INFO 4486 --- [alarm-detect-11,18faec680e765c5b,00ddda4d23b31723] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:30:46.926 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,0817daf2d8004661,de199f54928acc24] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0817daf2d8004661, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:46.926 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,0817daf2d8004661,de199f54928acc24] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0817daf2d8004661, duration=2 ms] 2024-10-16 13:30:46.994 INFO 4486 --- [prometheus-minute-schedule-1,9cca54818093704a,7f0593c10ef53429] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:47.000 INFO 4486 --- [prometheus-minute-schedule-1,9cca54818093704a,7f0593c10ef53429] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:47.012 INFO 4486 --- [ocp-async-4,a28b42252d3d9d0d,c79b615011013700] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:47.065 INFO 4486 --- [ocp-schedules-2,a443b1f8b3ffa0be,fedb110bf0540c3e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:30:47.070 INFO 4486 --- [ocp-schedules-5,9ed07f3e938c6e08,e9f23401a12c8401] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:30:47.179 INFO 4486 --- [ocp-schedules-24,b9abb4a505af44f5,0cd5991e2e144137] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:30:48.691 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:49.012 INFO 4486 --- [ocp-async-9,786722e056225d25,d97fe5edfaf30283] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:49.177 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d092101a74e1ee6e,63816327da92e6c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d092101a74e1ee6e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:49.177 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d092101a74e1ee6e,63816327da92e6c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d092101a74e1ee6e, duration=1 ms] 2024-10-16 13:30:49.711 INFO 4486 --- [ocp-schedules-3,8b6c882b98cd9495,97312a83b753a3ce] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:30:49.711 INFO 4486 --- [ocp-schedules-3,8b6c882b98cd9495,97312a83b753a3ce] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:30:49.711 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:30:49.711 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:30:49.711 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:30:49.711 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:30:49.711 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:30:49.717 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:30:49.717 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:49.717 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:30:49.718 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:30:49.718 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:30:49.719 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:30:49.719 INFO 4486 --- [ocp-schedules-3,8b6c882b98cd9495,97312a83b753a3ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:30:49.719 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:30:49.719 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:30:49.720 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:30:49.720 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:49.720 INFO 4486 --- [ocp-schedules-3,8b6c882b98cd9495,97312a83b753a3ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:30:49.743 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:30:49.749 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:30:49.749 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:30:49.756 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:30:49.757 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:30:49.785 INFO 4486 --- [ocp-schedules-3,8b6c882b98cd9495,97312a83b753a3ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:30:49.788 INFO 4486 --- [ocp-schedules-3,8b6c882b98cd9495,97312a83b753a3ce] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:30:49.994 INFO 4486 --- [prometheus-minute-schedule-1,b7bfd107bbc55f85,7013cef44da30156] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:49.999 INFO 4486 --- [prometheus-minute-schedule-1,b7bfd107bbc55f85,7013cef44da30156] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:50.201 INFO 4486 --- [ocp-schedules-14,11881f08291e79ad,2a254b3318e87760] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:30:50.353 INFO 4486 --- [ocp-schedules-5,b4be0fc2b0188467,fc4ae0c8ea0d6ed8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:30:50.353 INFO 4486 --- [ocp-schedules-5,b4be0fc2b0188467,fc4ae0c8ea0d6ed8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:30:50.355 INFO 4486 --- [ocp-schedules-5,b4be0fc2b0188467,fc4ae0c8ea0d6ed8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:30:50.355 INFO 4486 --- [ocp-schedules-5,b4be0fc2b0188467,fc4ae0c8ea0d6ed8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:30:50.355 INFO 4486 --- [ocp-schedules-5,b4be0fc2b0188467,fc4ae0c8ea0d6ed8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:30:50.355 INFO 4486 --- [ocp-schedules-5,b4be0fc2b0188467,fc4ae0c8ea0d6ed8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:30:50.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,93fd813c5c09a559,31c1e929330a46d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=93fd813c5c09a559, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:30:50.485 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,93fd813c5c09a559,31c1e929330a46d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=93fd813c5c09a559, duration=2 ms] 2024-10-16 13:30:50.531 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:30:50.531 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:30:50.545 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:30:50.545 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:30:50.545 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:30:50.557 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:30:50.557 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:30:50.557 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:30:50.563 INFO 4486 --- [ocp-schedules-27,97f16e5634e16f34,6bff73571684d6a9] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:30:50.762 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,902d5a21fbe9982a,5448b0cb784ca5b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=902d5a21fbe9982a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:50.762 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,902d5a21fbe9982a,5448b0cb784ca5b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=902d5a21fbe9982a, duration=1 ms] 2024-10-16 13:30:51.012 INFO 4486 --- [ocp-async-6,e8dce5fdb31bc887,7d377a0d1eae5a87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:51.151 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6db3aa53bb7e57ea,4920300a875544a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6db3aa53bb7e57ea, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:51.151 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6db3aa53bb7e57ea,4920300a875544a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6db3aa53bb7e57ea, duration=1 ms] 2024-10-16 13:30:51.692 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:52.983 INFO 4486 --- [ocp-async-1,58be4abb9a872156,6059a4f24bdd4169] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:30:52.994 INFO 4486 --- [prometheus-minute-schedule-1,129acb43df63803c,a37cbc470941d3d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:53.000 INFO 4486 --- [prometheus-minute-schedule-1,129acb43df63803c,a37cbc470941d3d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:53.012 INFO 4486 --- [ocp-async-4,51410a6262942a2e,da110d8dfd830f77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:53.445 INFO 4486 --- [ocp-schedules-12,1532d28566170c3d,27a1ca29b49408f2] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:30:53.447 INFO 4486 --- [ocp-schedules-12,1532d28566170c3d,27a1ca29b49408f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:53.448 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:30:53.448 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:30:53.448 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:30:54.693 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:55.013 INFO 4486 --- [ocp-async-3,6569460339eeff80,fd51baf54799824d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:55.994 INFO 4486 --- [prometheus-minute-schedule-1,c379ed0bcde63390,ee296ac06579126d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:55.999 INFO 4486 --- [prometheus-minute-schedule-1,c379ed0bcde63390,ee296ac06579126d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:56.366 INFO 4486 --- [ocp-schedules-23,d953a9eb3ce94bbf,7803a23d3dd1ba6a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:30:56.390 WARN 4486 --- [scanner-13242650-1,d953a9eb3ce94bbf,3ad4b2465dca2bd3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:30:56.453 INFO 4486 --- [alarm-detect-6,d953a9eb3ce94bbf,b61e98d807dbee21] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056630 ~ 1729056646 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056645, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056645, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056645=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:30:56.892 INFO 4486 --- [ocp-schedules-16,aed55b2b48b70833,a71093d831466719] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:30:56.892 INFO 4486 --- [ocp-schedules-16,079eeb1a9f44c804,0bb169cba5810ecd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:30:56.894 INFO 4486 --- [ocp-schedules-16,079eeb1a9f44c804,0bb169cba5810ecd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:30:56.927 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:56.927 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:56.931 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1b925856484eb8a3,1fb587bc84c3567c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1b925856484eb8a3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:56.932 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1b925856484eb8a3,1fb587bc84c3567c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1b925856484eb8a3, duration=2 ms] 2024-10-16 13:30:56.941 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:56.945 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:56.970 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:56.971 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:56.973 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:56.974 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:56.975 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:30:57.013 INFO 4486 --- [ocp-async-10,5e0f2dc7f4812cb9,b93611de33516e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:57.019 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:57.019 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.020 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:57.021 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:30:57.021 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.024 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:30:57.026 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:57.035 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.035 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:57.048 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.057 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:57.058 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.059 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:57.060 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:30:57.066 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:30:57.070 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:30:57.072 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:30:57.098 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:57.100 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.101 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:30:57.105 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:30:57.150 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:30:57.150 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:30:57.159 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:57.161 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.162 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:30:57.165 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.169 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:30:57.172 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:57.193 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:30:57.202 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.202 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:57.204 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:57.206 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.206 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:30:57.216 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:30:57.216 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:30:57.233 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.233 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:57.262 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.262 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:57.290 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.290 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:30:57.374 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.374 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:30:57.403 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:30:57.431 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:57.433 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.434 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:30:57.439 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:30:57.445 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:30:57.450 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:30:57.452 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:30:57.476 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:30:57.613 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:30:57.639 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:30:57.641 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:30:57.642 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:30:57.670 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:30:57.670 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:30:57.693 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:30:58.818 INFO 4486 --- [ocp-schedules-27,433c9622298fc6e9,3343013e6b2e9c4f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:30:58.994 INFO 4486 --- [prometheus-minute-schedule-1,1bfe8db6495342b1,2d5a1452af60958a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056660, interval=60 2024-10-16 13:30:58.999 INFO 4486 --- [prometheus-minute-schedule-1,1bfe8db6495342b1,2d5a1452af60958a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:30:59.013 INFO 4486 --- [ocp-async-1,84b7c6a1b8a7719d,c85075538d08dbbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:30:59.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d815f01a476fdf60,5f3c40d9b255e9ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d815f01a476fdf60, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:30:59.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d815f01a476fdf60,5f3c40d9b255e9ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d815f01a476fdf60, duration=2 ms] 2024-10-16 13:31:00.206 INFO 4486 --- [ocp-schedules-10,63d8b3d28d45dd34,66493ec229251a50] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:31:00.695 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:00.769 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b5a58abb6ebf4043,72b5f35131817223] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b5a58abb6ebf4043, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:00.769 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b5a58abb6ebf4043,72b5f35131817223] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b5a58abb6ebf4043, duration=1 ms] 2024-10-16 13:31:00.920 INFO 4486 --- [ocp-schedules-22,29aecf0e390ba9c2,b46b86abce56624c] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:31:00.920+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:31:00.923 INFO 4486 --- [ocp-schedules-22,29aecf0e390ba9c2,b46b86abce56624c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:31:00.928 INFO 4486 --- [ocp-schedules-22,29aecf0e390ba9c2,b46b86abce56624c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:31:00.939 INFO 4486 --- [ocp-schedules-22,29aecf0e390ba9c2,b46b86abce56624c] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:31:01.013 INFO 4486 --- [ocp-async-2,bb3de05a79a5ffb7,4e471f9d75444818] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:01.156 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e0b21e0d06273a1a,fbea1a74177b1a4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e0b21e0d06273a1a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:01.157 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e0b21e0d06273a1a,fbea1a74177b1a4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e0b21e0d06273a1a, duration=1 ms] 2024-10-16 13:31:01.994 INFO 4486 --- [prometheus-minute-schedule-1,5b170cbc38d755b7,cb36611e7b594d9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:01.999 INFO 4486 --- [prometheus-minute-schedule-1,5b170cbc38d755b7,cb36611e7b594d9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:02.983 INFO 4486 --- [ocp-async-9,75ae871f7d0effae,627f41c31692ddcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:31:03.013 INFO 4486 --- [ocp-async-5,ca815544a77ba417,52379ebbaa31b840] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:03.449 INFO 4486 --- [ocp-schedules-12,1532d28566170c3d,27a1ca29b49408f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:31:03.695 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:04.994 INFO 4486 --- [prometheus-minute-schedule-1,03232b39c12d8323,153e331fd514f478] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:05.001 INFO 4486 --- [prometheus-minute-schedule-1,03232b39c12d8323,153e331fd514f478] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:05.014 INFO 4486 --- [ocp-async-1,3ec5e0efeb9540ff,dd734d8931aef8c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:06.530 INFO 4486 --- [ocp-schedules-27,b25c3f6d2606273a,d4917efb817e4373] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:31:06.564 WARN 4486 --- [scanner-13242891-1,b25c3f6d2606273a,5fa6aef06a46a1ca] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:31:06.680 INFO 4486 --- [alarm-detect-20,b25c3f6d2606273a,1891fe5cd1d934b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:06.680 INFO 4486 --- [alarm-detect-2,b25c3f6d2606273a,f52364e1e275c4b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:06.681 INFO 4486 --- [alarm-detect-15,b25c3f6d2606273a,907a863a73051a08] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:06.684 INFO 4486 --- [alarm-detect-2,b25c3f6d2606273a,f52364e1e275c4b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:06.685 INFO 4486 --- [alarm-detect-15,b25c3f6d2606273a,907a863a73051a08] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:06.686 INFO 4486 --- [alarm-detect-20,b25c3f6d2606273a,1891fe5cd1d934b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:06.688 INFO 4486 --- [alarm-detect-2,b25c3f6d2606273a,f52364e1e275c4b6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:06.690 INFO 4486 --- [alarm-detect-15,b25c3f6d2606273a,907a863a73051a08] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:06.691 INFO 4486 --- [alarm-detect-20,b25c3f6d2606273a,1891fe5cd1d934b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:06.697 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:06.793 INFO 4486 --- [alarm-detect-4,b25c3f6d2606273a,b461010231b0dedb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:06.797 INFO 4486 --- [alarm-detect-4,b25c3f6d2606273a,b461010231b0dedb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:06.802 INFO 4486 --- [alarm-detect-4,b25c3f6d2606273a,b461010231b0dedb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:06.937 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4dcb8ccd55910fc7,8cbb9dc5397ce4a6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4dcb8ccd55910fc7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:06.937 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4dcb8ccd55910fc7,8cbb9dc5397ce4a6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4dcb8ccd55910fc7, duration=2 ms] 2024-10-16 13:31:07.014 INFO 4486 --- [ocp-async-7,673b057a8171c23b,d03bc4dc372a6c25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:07.066 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:31:07.067 INFO 4486 --- [ocp-schedules-30,4281cdcbdf0ce6cd,24385479d8b1ff91] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:31:07.068 INFO 4486 --- [ocp-schedules-30,4281cdcbdf0ce6cd,24385479d8b1ff91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:07.081 INFO 4486 --- [ocp-schedules-21,350a5204f4ad46cb,98a6828f19c2b67b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:31:07.083 INFO 4486 --- [ocp-schedules-21,350a5204f4ad46cb,98a6828f19c2b67b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:31:07.089 INFO 4486 --- [ocp-schedules-20,304ccf26bd19e219,9b70f59ed5860ebc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:31:07.090 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:31:07.090 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:31:07.090 INFO 4486 --- [ocp-schedules-5,8ea0441873feb5c2,b3c8abddc65285f5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:31:07.090 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:31:07.091 INFO 4486 --- [ocp-schedules-7,9f608996e52a3d0a,59d2c56a7da120d9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:31:07.091 INFO 4486 --- [host-check-clockdiff1,8ea0441873feb5c2,fb432b0678ded864] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:31:07.091 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:31:07.091 INFO 4486 --- [host-check-clockdiff2,8ea0441873feb5c2,d3da2a5e2404a721] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:31:07.091 INFO 4486 --- [host-check-clockdiff3,8ea0441873feb5c2,84e2b32b20a85d93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:31:07.092 INFO 4486 --- [host-check-clockdiff4,8ea0441873feb5c2,8b12976c568b2654] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:31:07.092 INFO 4486 --- [host-check-agent-status3,9f608996e52a3d0a,2488866c9032a7cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:31:07.092 INFO 4486 --- [host-check-agent-status1,9f608996e52a3d0a,0b95bf591e58a0ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:31:07.092 INFO 4486 --- [host-check-agent-status2,9f608996e52a3d0a,0d45ffc734559de4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:31:07.093 INFO 4486 --- [host-check-agent-status4,9f608996e52a3d0a,bc7cd68d01eb09c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:31:07.097 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:07.098 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:31:07.098 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:31:07.098 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:31:07.099 INFO 4486 --- [host-check-agent-status2,9f608996e52a3d0a,0d45ffc734559de4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:31:07.099 INFO 4486 --- [host-check-agent-status4,9f608996e52a3d0a,bc7cd68d01eb09c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:31:07.099 INFO 4486 --- [host-check-agent-status1,9f608996e52a3d0a,0b95bf591e58a0ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:07.099 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:07.100 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:31:07.100 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:31:07.100 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:31:07.101 INFO 4486 --- [host-check-agent-status1,9f608996e52a3d0a,0b95bf591e58a0ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:07.101 INFO 4486 --- [host-check-agent-status4,9f608996e52a3d0a,bc7cd68d01eb09c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:31:07.101 INFO 4486 --- [host-check-agent-status2,9f608996e52a3d0a,0d45ffc734559de4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:31:07.102 INFO 4486 --- [host-check-agent-status3,9f608996e52a3d0a,2488866c9032a7cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:31:07.108 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:07.125 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.126 INFO 4486 --- [host-check-agent-status3,9f608996e52a3d0a,2488866c9032a7cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:31:07.130 INFO 4486 --- [host-check-clockdiff1,8ea0441873feb5c2,fb432b0678ded864] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056667 0 0 , error= 2024-10-16 13:31:07.130 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:07.134 INFO 4486 --- [host-check-clockdiff3,8ea0441873feb5c2,84e2b32b20a85d93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056667 3 3 , error= 2024-10-16 13:31:07.135 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:31:07.154 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:07.154 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:31:07.154 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:31:07.155 INFO 4486 --- [host-check-agent-status2,9f608996e52a3d0a,0d45ffc734559de4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:31:07.156 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.157 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.157 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:31:07.157 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:31:07.158 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:31:07.160444062+08:00 2024-10-16 13:31:07.158 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:31:07.159936804+08:00 2024-10-16 13:31:07.158 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:07.158 INFO 4486 --- [host-check-agent-status4,9f608996e52a3d0a,bc7cd68d01eb09c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:31:07.158 INFO 4486 --- [host-check-agent-status1,9f608996e52a3d0a,0b95bf591e58a0ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:31:07.158 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:31:07.158+08:00 2024-10-16 13:31:07.158 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:31:07.158+08:00 2024-10-16 13:31:07.160 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:31:07.159660693+08:00 2024-10-16 13:31:07.160 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:31:07.162563746+08:00 2024-10-16 13:31:07.160 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:31:07.160+08:00 2024-10-16 13:31:07.160 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:31:07.160+08:00 2024-10-16 13:31:07.161 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:31:07.161 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:31:07.158+08:00 and status AVAILABLE 2024-10-16 13:31:07.162 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:31:07.158+08:00 and status AVAILABLE 2024-10-16 13:31:07.162 INFO 4486 --- [host-check-agent-status2,9f608996e52a3d0a,0d45ffc734559de4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:31:07.162 INFO 4486 --- [host-check-agent-status2,9f608996e52a3d0a,0d45ffc734559de4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:31:07.162 INFO 4486 --- [host-check-agent-status2,9f608996e52a3d0a,0d45ffc734559de4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:31:07.162 INFO 4486 --- [host-check-agent-status3,9f608996e52a3d0a,2488866c9032a7cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:31:07.163 INFO 4486 --- [host-check-agent-status4,9f608996e52a3d0a,bc7cd68d01eb09c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:31:07.163 INFO 4486 --- [host-check-agent-status4,9f608996e52a3d0a,bc7cd68d01eb09c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:31:07.163 INFO 4486 --- [host-check-agent-status4,9f608996e52a3d0a,bc7cd68d01eb09c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:31:07.163 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:31:07.164 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:31:07.160+08:00 and status AVAILABLE 2024-10-16 13:31:07.164 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:31:07.160+08:00 and status AVAILABLE 2024-10-16 13:31:07.164 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:31:07.164 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:31:07.166 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:31:07.166 INFO 4486 --- [host-check-agent-status3,9f608996e52a3d0a,2488866c9032a7cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:31:07.166 INFO 4486 --- [host-check-agent-status3,9f608996e52a3d0a,2488866c9032a7cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:31:07.166 INFO 4486 --- [host-check-agent-status3,9f608996e52a3d0a,2488866c9032a7cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:31:07.167 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:31:07.169 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.170 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:31:07.175 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:31:07.175 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:31:07.175 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:07.176 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:31:07.176 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:31:07.176 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:31:07.177 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:07.177 INFO 4486 --- [host-check-agent-status1,9f608996e52a3d0a,0b95bf591e58a0ac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:31:07.177 INFO 4486 --- [host-check-agent-status1,9f608996e52a3d0a,0b95bf591e58a0ac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:31:07.177 INFO 4486 --- [host-check-agent-status1,9f608996e52a3d0a,0b95bf591e58a0ac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:31:07.178 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:31:07.182 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.183 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:31:07.193 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.194 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:31:07.196 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.197 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:31:07.199 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:07.199 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:07.200 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:07.201 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:07.209 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:07.210 INFO 4486 --- [ocp-schedules-29,d0b61ed1180a097f,8622e4fe258f81fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:31:07.229 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:31:07.234 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:31:07.239 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:31:07.244 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:31:07.246 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:31:07.251 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:31:07.256 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:31:07.256 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:07.261 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:31:07.265 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:07.270 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:07.276 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:07.284 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:31:07.293 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:31:07.299 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:31:07.309 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:31:07.310 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:07.310 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:31:07.321 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:31:07.326 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:07.339 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:07.339 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:31:07.340 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:31:07.350 INFO 4486 --- [host-check-host-status3,304ccf26bd19e219,56d343babd821b73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:31:07.352 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:31:07.361 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:31:07.372 INFO 4486 --- [host-check-host-status4,304ccf26bd19e219,445269a2d0c89946] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:31:07.373 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:07.375 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:31:07.385 INFO 4486 --- [host-check-host-status2,304ccf26bd19e219,660402d00131cb9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:31:07.430 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:31:07.441 INFO 4486 --- [host-check-host-status1,304ccf26bd19e219,c4e5ca289e40f962] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:31:07.994 INFO 4486 --- [prometheus-minute-schedule-1,539a37fe55efa6c7,4cf0bb4927d9a562] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:08.000 INFO 4486 --- [prometheus-minute-schedule-1,539a37fe55efa6c7,4cf0bb4927d9a562] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:08.567 INFO 4486 --- [ocp-schedules-4,38926ef270315e23,cf1586a60dbae4bf] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:31:08.569 INFO 4486 --- [ocp-schedules-4,38926ef270315e23,cf1586a60dbae4bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:09.014 INFO 4486 --- [ocp-async-10,2f6e8be783da9adc,9a4a3c98782fefed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:09.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a2383f0043659b2f,47b46f155c5d4f63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a2383f0043659b2f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:09.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a2383f0043659b2f,47b46f155c5d4f63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a2383f0043659b2f, duration=0 ms] 2024-10-16 13:31:09.636 INFO 4486 --- [host-check-clockdiff4,8ea0441873feb5c2,8b12976c568b2654] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056669 3 3 , error= 2024-10-16 13:31:09.698 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:10.006 INFO 4486 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:31:10.006 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:31:10.029 INFO 4486 --- [Check obproxy1,34e4328e7f6768af,97d2a0cde517262a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:31:10.029 INFO 4486 --- [Check obproxy1,34e4328e7f6768af,97d2a0cde517262a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:10.029 INFO 4486 --- [Check obproxy2,c91daf7fb9d13d2f,0ec68773db7a4581] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:31:10.029 INFO 4486 --- [Check obproxy2,c91daf7fb9d13d2f,0ec68773db7a4581] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:10.031 INFO 4486 --- [Collect obproxy2,c3d550bd70ec77ef,f7bc296a3efb867c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:31:10.031 INFO 4486 --- [Collect obproxy2,c3d550bd70ec77ef,f7bc296a3efb867c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:10.033 INFO 4486 --- [Collect obproxy1,068f9b2ac4afb3fc,2c10885f9ecdd602] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:31:10.033 INFO 4486 --- [Collect obproxy1,068f9b2ac4afb3fc,2c10885f9ecdd602] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:10.051 INFO 4486 --- [Check obproxy1,34e4328e7f6768af,97d2a0cde517262a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:31:10.051 INFO 4486 --- [Check obproxy1,34e4328e7f6768af,97d2a0cde517262a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:31:10.060 INFO 4486 --- [Check obproxy2,c91daf7fb9d13d2f,0ec68773db7a4581] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:31:10.060 INFO 4486 --- [Check obproxy2,c91daf7fb9d13d2f,0ec68773db7a4581] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:31:10.063 INFO 4486 --- [Collect obproxy2,c3d550bd70ec77ef,f7bc296a3efb867c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:31:10.067 INFO 4486 --- [Collect obproxy2,c3d550bd70ec77ef,f7bc296a3efb867c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:31:10.068 INFO 4486 --- [Collect obproxy1,068f9b2ac4afb3fc,2c10885f9ecdd602] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:31:10.070 INFO 4486 --- [Collect obproxy2,c3d550bd70ec77ef,f7bc296a3efb867c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:31:10.071 INFO 4486 --- [Collect obproxy1,068f9b2ac4afb3fc,2c10885f9ecdd602] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:31:10.074 INFO 4486 --- [Collect obproxy1,068f9b2ac4afb3fc,2c10885f9ecdd602] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:31:10.213 INFO 4486 --- [ocp-schedules-22,d950193b5846e995,6dea83289f4d723d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:31:10.773 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,aaf65358fc59401d,89d70d525feb4f53] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=aaf65358fc59401d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:10.774 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,aaf65358fc59401d,89d70d525feb4f53] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=aaf65358fc59401d, duration=2 ms] 2024-10-16 13:31:10.802 INFO 4486 --- [host-check-clockdiff2,8ea0441873feb5c2,d3da2a5e2404a721] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056670 3 3 , error= 2024-10-16 13:31:10.977 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:31:10.994 INFO 4486 --- [prometheus-minute-schedule-1,594d58d6234d579e,20edf76355332106] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:10.999 INFO 4486 --- [prometheus-minute-schedule-1,594d58d6234d579e,20edf76355332106] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:11.014 INFO 4486 --- [ocp-async-8,5bfd986805be5178,7106a4e5c5fde08e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:11.015 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:31:11.160 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e244135b6455a0af,f86869d03b6a6939] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e244135b6455a0af, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:11.160 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e244135b6455a0af,f86869d03b6a6939] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e244135b6455a0af, duration=1 ms] 2024-10-16 13:31:12.699 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:12.983 INFO 4486 --- [ocp-async-1,4bab77098a3e350f,068ac98c094292e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:31:13.027 ERROR 4486 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620709) Duplicate entry '586679390270197760-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.005939] [YB420AC531B8-00060B7DB419572B-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620709) Duplicate entry '586679390270197760-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.005939] [YB420AC531B8-00060B7DB419572B-0-0] 2024-10-16 13:31:13.027 INFO 4486 --- [ocp-async-2,aa4f655cc1704578,271946387c80b1b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:13.027 ERROR 4486 --- [metric-archive-second-27,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990814) Duplicate entry '669741696490737664-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.005419] [YB420AC531B8-00060B7DB46FA57E-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990814) Duplicate entry '669741696490737664-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.005419] [YB420AC531B8-00060B7DB46FA57E-0-0] 2024-10-16 13:31:13.027 ERROR 4486 --- [metric-archive-second-19,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2457395) Duplicate entry '586774769061924864-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.020704] [YB420AC531B8-00060B7DA1ABD5F5-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2457395) Duplicate entry '586774769061924864-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.020704] [YB420AC531B8-00060B7DA1ABD5F5-0-0] 2024-10-16 13:31:13.027 ERROR 4486 --- [metric-archive-second-28,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=4117402) Duplicate entry '669741696503320576-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.010594] [YB420AC531B8-00060B7D84EAAC46-0-0]; nested exception is java.sql.BatchUpdateException: (conn=4117402) Duplicate entry '669741696503320576-1729056600' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:31:13.010594] [YB420AC531B8-00060B7D84EAAC46-0-0] 2024-10-16 13:31:13.489 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056600, nodeCount=787, affectRows=787 2024-10-16 13:31:13.495 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056600, nodeCount=786, validCount=786 2024-10-16 13:31:13.551 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056600, nodeCount=816, affectRows=816 2024-10-16 13:31:13.553 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056600, nodeCount=916, affectRows=916 2024-10-16 13:31:13.556 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056600, nodeCount=816, validCount=816 2024-10-16 13:31:13.558 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056600, nodeCount=915, validCount=915 2024-10-16 13:31:13.601 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056600, nodeCount=931, affectRows=931 2024-10-16 13:31:13.607 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056600, nodeCount=930, validCount=930 2024-10-16 13:31:13.994 INFO 4486 --- [prometheus-minute-schedule-1,f63713ce664b424b,1fe2a8096e248e4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:14.000 INFO 4486 --- [prometheus-minute-schedule-1,f63713ce664b424b,1fe2a8096e248e4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:14.367 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056600, nodeCount=480, affectRows=480 2024-10-16 13:31:15.027 INFO 4486 --- [ocp-async-9,42f361f8adbd93a7,cf5a6f33d71724c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:15.088 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056600, nodeCount=480, validCount=480 2024-10-16 13:31:15.700 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:16.832 INFO 4486 --- [ocp-schedules-19,9577176d81eec704,8aeaa51332bcc0e0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:31:16.872 WARN 4486 --- [scanner-13244042-1,9577176d81eec704,f468046845f3ac23] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:31:16.929 INFO 4486 --- [alarm-detect-9,9577176d81eec704,0538b997b8f14937] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056645 ~ 1729056666 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056660, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056660, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056660=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:31:16.962 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,74bfcd02d6b964b3,da4a488c61d105f4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=74bfcd02d6b964b3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:16.963 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,74bfcd02d6b964b3,da4a488c61d105f4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=74bfcd02d6b964b3, duration=16 ms] 2024-10-16 13:31:16.994 INFO 4486 --- [prometheus-minute-schedule-1,a99d9a533df14cc0,e1a643904d7528a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:16.999 INFO 4486 --- [prometheus-minute-schedule-1,a99d9a533df14cc0,e1a643904d7528a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:17.027 INFO 4486 --- [ocp-async-5,457ab15315ea6984,984198150ce1bf0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:17.068 INFO 4486 --- [ocp-schedules-7,1c121836806f2ac1,93ec637fd5bd17ca] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:31:17.070 INFO 4486 --- [ocp-schedules-30,4281cdcbdf0ce6cd,24385479d8b1ff91] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:31:17.164 INFO 4486 --- [ocp-schedules-21,350a5204f4ad46cb,98a6828f19c2b67b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:31:18.185 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,35efa10dc53ef54b,079000b150cf68f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=35efa10dc53ef54b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:31:18.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,35efa10dc53ef54b,079000b150cf68f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=35efa10dc53ef54b, duration=2 ms] 2024-10-16 13:31:18.599 INFO 4486 --- [ocp-schedules-4,38926ef270315e23,cf1586a60dbae4bf] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:31:18.701 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:18.823 INFO 4486 --- [ocp-schedules-7,2cebf456e6f52357,abc624312e13fd35] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:31:19.028 INFO 4486 --- [ocp-async-1,a8da4b23e0579337,faed508c9c44722b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:19.194 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc0d73ad5fec3676,dab652760f96247b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cc0d73ad5fec3676, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:19.194 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc0d73ad5fec3676,dab652760f96247b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cc0d73ad5fec3676, duration=1 ms] 2024-10-16 13:31:19.994 INFO 4486 --- [prometheus-minute-schedule-1,c2053c251445b578,fd26f1509da6b515] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:20.000 INFO 4486 --- [prometheus-minute-schedule-1,c2053c251445b578,fd26f1509da6b515] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:20.218 INFO 4486 --- [ocp-schedules-9,df5805ba4bc99ea1,89741678745ae8d4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:31:20.779 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cab1f8a01c06bd96,32fe9cdad20567f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cab1f8a01c06bd96, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:20.780 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cab1f8a01c06bd96,32fe9cdad20567f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cab1f8a01c06bd96, duration=2 ms] 2024-10-16 13:31:20.996 INFO 4486 --- [prometheus-cache-monitor1,680c28ace8f92126,cd6e36c9c55f5fcc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:31:20.997 INFO 4486 --- [prometheus-cache-monitor1,680c28ace8f92126,cd6e36c9c55f5fcc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:31:21.028 INFO 4486 --- [ocp-async-3,56c551e19d58f24e,ea3298409c66eb09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:21.165 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,8527691bfe187efd,d1dd5450cf353ef7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8527691bfe187efd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:21.165 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,8527691bfe187efd,d1dd5450cf353ef7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8527691bfe187efd, duration=1 ms] 2024-10-16 13:31:21.702 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:22.661 INFO 4486 --- [ocp-schedules-21,3f8ba277cef9857b,6dda37b9302ec104] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:31:22.984 INFO 4486 --- [ocp-async-9,d29194075c58927e,a19abc1694de7f87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:31:22.994 INFO 4486 --- [prometheus-minute-schedule-1,3bc5fad77dc2a599,348bf4ec38bcaeaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:22.999 INFO 4486 --- [prometheus-minute-schedule-1,3bc5fad77dc2a599,348bf4ec38bcaeaf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:23.028 INFO 4486 --- [ocp-async-6,60217a2dba823d12,20700673d0ca567f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:23.634 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a612f64799b9090e,4f6a50e4834008f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=a612f64799b9090e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:31:23.634 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a612f64799b9090e,4f6a50e4834008f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=a612f64799b9090e, duration=1 ms] 2024-10-16 13:31:23.736 INFO 4486 --- [ocp-schedules-26,ed14f533f1405471,eb00bb7a60e034a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:31:24.101 INFO 4486 --- [ocp-schedules-22,2c56bef9f32f1de4,fbe1e23dd4e59f25] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:31:24.304 INFO 4486 --- [ocp-schedules-25,36be65bdc17b87ea,9f62a99caa37eb41] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:31:24.304 INFO 4486 --- [ocp-schedules-25,36be65bdc17b87ea,9f62a99caa37eb41] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:31:24.335 INFO 4486 --- [Collect all tenants major compactions1,36be65bdc17b87ea,fceb03c7eab26bad] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:31:24.335 INFO 4486 --- [Collect all tenants major compactions2,36be65bdc17b87ea,0e5e9252e5dad064] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:31:24.335 INFO 4486 --- [Collect all tenants major compactions4,36be65bdc17b87ea,d5538a17cd9f28f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:31:24.335 INFO 4486 --- [Collect all tenants major compactions3,36be65bdc17b87ea,c38e0fd8aca83159] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:31:24.337 INFO 4486 --- [Collect all tenants major compactions5,36be65bdc17b87ea,9cd37007b7737012] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:31:24.348 INFO 4486 --- [Collect all tenants major compactions1,36be65bdc17b87ea,fceb03c7eab26bad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:24.348 INFO 4486 --- [Collect all tenants major compactions2,36be65bdc17b87ea,0e5e9252e5dad064] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:24.350 INFO 4486 --- [Collect all tenants major compactions1,36be65bdc17b87ea,fceb03c7eab26bad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:31:24.350 INFO 4486 --- [Collect all tenants major compactions2,36be65bdc17b87ea,0e5e9252e5dad064] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:31:24.353 INFO 4486 --- [Collect all tenants major compactions2,36be65bdc17b87ea,0e5e9252e5dad064] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:31:24.353 INFO 4486 --- [Collect all tenants major compactions1,36be65bdc17b87ea,fceb03c7eab26bad] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:31:24.353 INFO 4486 --- [Collect all tenants major compactions2,36be65bdc17b87ea,0e5e9252e5dad064] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:31:24.354 INFO 4486 --- [Collect all tenants major compactions1,36be65bdc17b87ea,fceb03c7eab26bad] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:31:24.355 INFO 4486 --- [Collect all tenants major compactions3,36be65bdc17b87ea,c38e0fd8aca83159] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:24.357 INFO 4486 --- [Collect all tenants major compactions3,36be65bdc17b87ea,c38e0fd8aca83159] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:31:24.360 INFO 4486 --- [Collect all tenants major compactions3,36be65bdc17b87ea,c38e0fd8aca83159] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:31:24.360 INFO 4486 --- [Collect all tenants major compactions3,36be65bdc17b87ea,c38e0fd8aca83159] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:31:24.365 INFO 4486 --- [Collect all tenants major compactions4,36be65bdc17b87ea,d5538a17cd9f28f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:24.367 INFO 4486 --- [Collect all tenants major compactions4,36be65bdc17b87ea,d5538a17cd9f28f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:31:24.369 INFO 4486 --- [Collect all tenants major compactions5,36be65bdc17b87ea,9cd37007b7737012] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:24.370 INFO 4486 --- [Collect all tenants major compactions4,36be65bdc17b87ea,d5538a17cd9f28f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:31:24.370 INFO 4486 --- [Collect all tenants major compactions4,36be65bdc17b87ea,d5538a17cd9f28f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:31:24.372 INFO 4486 --- [Collect all tenants major compactions5,36be65bdc17b87ea,9cd37007b7737012] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:31:24.377 INFO 4486 --- [Collect all tenants major compactions5,36be65bdc17b87ea,9cd37007b7737012] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:31:24.377 INFO 4486 --- [Collect all tenants major compactions5,36be65bdc17b87ea,9cd37007b7737012] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:31:24.379 INFO 4486 --- [ocp-schedules-25,36be65bdc17b87ea,9f62a99caa37eb41] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:31:24.703 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:25.028 INFO 4486 --- [ocp-async-7,d3328a0200c0a09a,064b4652efedaa15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:25.994 INFO 4486 --- [prometheus-minute-schedule-1,e3a695640ab8b990,29ae9ca6c1c595fc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:26.000 INFO 4486 --- [prometheus-minute-schedule-1,e3a695640ab8b990,29ae9ca6c1c595fc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:26.785 INFO 4486 --- [ocp-schedules-23,ecd4dbfeffe97cae,7cee2a12ebc83c13] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:31:26.785+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:31:26.787 INFO 4486 --- [ocp-schedules-23,ecd4dbfeffe97cae,7cee2a12ebc83c13] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:31:26.788 INFO 4486 --- [ocp-schedules-23,ecd4dbfeffe97cae,7cee2a12ebc83c13] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:31:26.788 INFO 4486 --- [ocp-schedules-23,ecd4dbfeffe97cae,7cee2a12ebc83c13] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:31:26.973 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,00507e87ff019e81,319cd16eeaaf7a7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=00507e87ff019e81, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:26.973 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,00507e87ff019e81,319cd16eeaaf7a7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=00507e87ff019e81, duration=1 ms] 2024-10-16 13:31:26.990 INFO 4486 --- [ocp-schedules-14,583ca859254d6ff6,f781a245656fd149] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:31:27.029 INFO 4486 --- [ocp-async-2,39188536aa7a0156,d0b40a355de205d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:27.035 WARN 4486 --- [scanner-13244265-1,583ca859254d6ff6,515f7d9e71d7ecd9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:31:27.053 INFO 4486 --- [alarm-detect-6,583ca859254d6ff6,494c4e83e63dda50] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:27.057 INFO 4486 --- [alarm-detect-6,583ca859254d6ff6,494c4e83e63dda50] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:27.061 INFO 4486 --- [alarm-detect-6,583ca859254d6ff6,494c4e83e63dda50] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:27.083 INFO 4486 --- [alarm-detect-1,583ca859254d6ff6,2bda3f12931468d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:27.083 INFO 4486 --- [alarm-detect-12,583ca859254d6ff6,bfd411cba5fa8629] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:27.087 INFO 4486 --- [alarm-detect-1,583ca859254d6ff6,2bda3f12931468d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:27.088 INFO 4486 --- [alarm-detect-12,583ca859254d6ff6,bfd411cba5fa8629] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:27.092 INFO 4486 --- [alarm-detect-12,583ca859254d6ff6,bfd411cba5fa8629] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:27.092 INFO 4486 --- [alarm-detect-1,583ca859254d6ff6,2bda3f12931468d3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:27.123 INFO 4486 --- [alarm-detect-15,583ca859254d6ff6,fde6fe894395aeea] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056660 ~ 1729056677 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056675, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056675, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056675=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:31:27.168 INFO 4486 --- [alarm-detect-1,583ca859254d6ff6,5badc54321f79019] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:27.175 INFO 4486 --- [alarm-detect-1,583ca859254d6ff6,5badc54321f79019] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:27.181 INFO 4486 --- [alarm-detect-1,583ca859254d6ff6,5badc54321f79019] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:27.671 INFO 4486 --- [ocp-schedules-16,079eeb1a9f44c804,0bb169cba5810ecd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:31:27.671 INFO 4486 --- [ocp-schedules-16,7106e1401f1e7071,af7862705a830755] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:31:27.673 INFO 4486 --- [ocp-schedules-16,7106e1401f1e7071,af7862705a830755] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:27.704 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:27.710 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.710 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:27.714 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.716 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:27.748 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.749 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:27.752 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:27.753 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.754 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:31:27.791 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:27.791 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.792 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:27.793 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:31:27.793 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.796 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:31:27.798 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:27.810 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:27.810 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:27.820 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:27.831 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:27.831 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.832 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:27.833 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:31:27.838 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:31:27.844 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:27.846 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:27.865 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:27.867 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.868 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:31:27.868 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:31:27.908 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:31:27.909 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:31:27.916 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:27.917 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.918 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:27.922 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.925 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:31:27.928 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:27.947 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:31:27.955 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:27.955 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:27.957 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:27.958 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:27.959 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:31:27.969 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:31:27.969 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:31:27.980 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:27.981 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:28.002 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:28.002 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:28.026 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:28.026 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:28.049 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:28.049 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:28.074 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:28.095 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:28.096 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:28.097 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:28.102 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:31:28.106 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:31:28.111 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:31:28.113 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:31:28.135 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:31:28.262 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:31:28.293 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:28.295 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:28.296 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:31:28.325 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:31:28.325 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:31:28.994 INFO 4486 --- [prometheus-minute-schedule-1,20fc1dc374c2b592,f37afedfb6755912] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:29.001 INFO 4486 --- [prometheus-minute-schedule-1,20fc1dc374c2b592,f37afedfb6755912] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:29.029 INFO 4486 --- [ocp-async-10,6941c5c7ba2b5491,a0ccb3d89fe11da1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:29.199 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f8de92002808fc10,475de7ac862acb91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f8de92002808fc10, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:29.199 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f8de92002808fc10,475de7ac862acb91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f8de92002808fc10, duration=2 ms] 2024-10-16 13:31:30.139 INFO 4486 --- [prometheus-second-schedule-1,d4f48887288e6a28,6e7742f8880e4332] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:31:30.222 INFO 4486 --- [ocp-schedules-18,5320104b1edf06cb,18b491e18ab07b0c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:31:30.704 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:30.785 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,80eedf424ed55ef7,42284db54b6bb4a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=80eedf424ed55ef7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:30.785 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,80eedf424ed55ef7,42284db54b6bb4a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=80eedf424ed55ef7, duration=1 ms] 2024-10-16 13:31:31.029 INFO 4486 --- [ocp-async-6,0369fff712347714,5dadd151410b7c8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:31.169 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,efe50fded8e2915a,ece1a20783fa61dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=efe50fded8e2915a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:31.169 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,efe50fded8e2915a,ece1a20783fa61dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=efe50fded8e2915a, duration=1 ms] 2024-10-16 13:31:31.994 INFO 4486 --- [prometheus-minute-schedule-1,5671ebf6fad0f85f,465d89435cefe4e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:31.999 INFO 4486 --- [prometheus-minute-schedule-1,5671ebf6fad0f85f,465d89435cefe4e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:32.984 INFO 4486 --- [ocp-async-7,123a002a1ce159ec,b52c0824cbc23ade] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:31:33.029 INFO 4486 --- [ocp-async-4,445f9968e3c90d24,4ca6b3b13319f483] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:33.705 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:34.249 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cd5d36e2ea378efa,dd47d4fe4f515bdb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=cd5d36e2ea378efa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:31:34.249 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cd5d36e2ea378efa,dd47d4fe4f515bdb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=cd5d36e2ea378efa, duration=1 ms] 2024-10-16 13:31:34.994 INFO 4486 --- [prometheus-minute-schedule-1,b073eb121ff2733b,b779426aeb6d05a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:34.999 INFO 4486 --- [prometheus-minute-schedule-1,b073eb121ff2733b,b779426aeb6d05a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:35.029 INFO 4486 --- [ocp-async-9,2a24e24411c1ed26,eba43359d5c4ef01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:36.706 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:36.980 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bc3b04dd7f695491,680321bd366b7f48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bc3b04dd7f695491, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:36.980 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bc3b04dd7f695491,680321bd366b7f48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bc3b04dd7f695491, duration=2 ms] 2024-10-16 13:31:37.029 INFO 4486 --- [ocp-async-1,ba169f4652f64d38,8845fea7adb4b7de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:37.063 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:31:37.064 INFO 4486 --- [ocp-schedules-13,b855eed81ed5161b,494d2585e038c8cc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:31:37.065 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:31:37.066 INFO 4486 --- [ocp-schedules-18,2057c636234bb05e,4cf58874aa36ffd0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:31:37.068 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:31:37.069 INFO 4486 --- [ocp-schedules-18,2057c636234bb05e,4cf58874aa36ffd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:37.070 INFO 4486 --- [ocp-schedules-13,b855eed81ed5161b,494d2585e038c8cc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:31:37.081 INFO 4486 --- [ocp-schedules-20,f2a67ec18c578f1b,09bd5f268ef7a305] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:31:37.083 INFO 4486 --- [ocp-schedules-20,f2a67ec18c578f1b,09bd5f268ef7a305] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:37.084 INFO 4486 --- [ocp-schedules-30,a4e4eb383e6cb8bc,8e9328d3584cf82a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 13:31:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:31:37.086 INFO 4486 --- [ocp-schedules-30,a4e4eb383e6cb8bc,8e9328d3584cf82a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:37.089 INFO 4486 --- [ocp-schedules-5,a1adefa9ba276d36,7b652bd207d92c76] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:31:37.090 INFO 4486 --- [host-check-agent-status1,a1adefa9ba276d36,b82061907bd22d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:31:37.090 INFO 4486 --- [host-check-agent-status2,a1adefa9ba276d36,6f7e2f14751d16d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:31:37.091 INFO 4486 --- [ocp-schedules-25,f2fee5a76bd4cae0,088eea3a0c9eeecd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:31:37.091 INFO 4486 --- [host-check-agent-status3,a1adefa9ba276d36,32b10f1c10964893] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:31:37.091 INFO 4486 --- [host-check-agent-status4,a1adefa9ba276d36,be8db65369c241b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:31:37.095 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:31:37.095 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:31:37.095 INFO 4486 --- [ocp-schedules-24,725aa09bb1cfe70a,1c8fdd1c5ea28bd6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:31:37.095 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:31:37.095 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:31:37.099 INFO 4486 --- [host-check-clockdiff3,725aa09bb1cfe70a,a6725dc02b130589] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:31:37.100 INFO 4486 --- [host-check-clockdiff1,725aa09bb1cfe70a,439a29a031b49bfd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:31:37.100 INFO 4486 --- [host-check-clockdiff4,725aa09bb1cfe70a,3aca68a6ab9c6046] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:31:37.100 INFO 4486 --- [host-check-clockdiff2,725aa09bb1cfe70a,cca106d417e2a5e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:31:37.101 INFO 4486 --- [host-check-agent-status1,a1adefa9ba276d36,b82061907bd22d1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:37.103 INFO 4486 --- [host-check-agent-status1,a1adefa9ba276d36,b82061907bd22d1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:37.104 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:31:37.104 INFO 4486 --- [host-check-agent-status4,a1adefa9ba276d36,be8db65369c241b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:31:37.104 INFO 4486 --- [host-check-agent-status2,a1adefa9ba276d36,6f7e2f14751d16d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:31:37.104 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:31:37.105 INFO 4486 --- [host-check-agent-status3,a1adefa9ba276d36,32b10f1c10964893] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:31:37.107 INFO 4486 --- [host-check-agent-status2,a1adefa9ba276d36,6f7e2f14751d16d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:31:37.107 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:31:37.107 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:31:37.107 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:37.108 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:31:37.108 INFO 4486 --- [host-check-agent-status4,a1adefa9ba276d36,be8db65369c241b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:31:37.108 INFO 4486 --- [host-check-agent-status3,a1adefa9ba276d36,32b10f1c10964893] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:31:37.125 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:37.127 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:31:37.127 INFO 4486 --- [host-check-clockdiff1,725aa09bb1cfe70a,439a29a031b49bfd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056697 0 0 , error= 2024-10-16 13:31:37.153 INFO 4486 --- [host-check-clockdiff2,725aa09bb1cfe70a,cca106d417e2a5e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056697 3 3 , error= 2024-10-16 13:31:37.160 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.160 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:37.161 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:31:37.165 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.166 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:31:37.179 INFO 4486 --- [host-check-agent-status2,a1adefa9ba276d36,6f7e2f14751d16d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:31:37.179 INFO 4486 --- [host-check-agent-status1,a1adefa9ba276d36,b82061907bd22d1e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:31:37.181 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:31:37.181 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:31:37.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,445a3dd1a9ceb5a1,68c939a803a164a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=445a3dd1a9ceb5a1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 13:31:37.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,445a3dd1a9ceb5a1,68c939a803a164a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=445a3dd1a9ceb5a1, duration=1 ms] 2024-10-16 13:31:37.183 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.183 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:31:37.186021713+08:00 2024-10-16 13:31:37.183 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:31:37.186092395+08:00 2024-10-16 13:31:37.183 INFO 4486 --- [host-check-agent-status3,a1adefa9ba276d36,32b10f1c10964893] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:31:37.184 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:31:37.183+08:00 2024-10-16 13:31:37.184 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:31:37.183+08:00 2024-10-16 13:31:37.184 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:37.185 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.185 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:31:37.185 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:31:37.185 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:31:37.186 INFO 4486 --- [host-check-agent-status1,a1adefa9ba276d36,b82061907bd22d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:31:37.186 INFO 4486 --- [host-check-agent-status2,a1adefa9ba276d36,6f7e2f14751d16d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:31:37.186 INFO 4486 --- [host-check-agent-status2,a1adefa9ba276d36,6f7e2f14751d16d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:31:37.186 INFO 4486 --- [host-check-agent-status1,a1adefa9ba276d36,b82061907bd22d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:31:37.186 INFO 4486 --- [host-check-agent-status2,a1adefa9ba276d36,6f7e2f14751d16d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:31:37.186 INFO 4486 --- [host-check-agent-status1,a1adefa9ba276d36,b82061907bd22d1e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:31:37.187 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:31:37.186992499+08:00 2024-10-16 13:31:37.187 INFO 4486 --- [host-check-agent-status3,a1adefa9ba276d36,32b10f1c10964893] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:31:37.187 INFO 4486 --- [host-check-agent-status3,a1adefa9ba276d36,32b10f1c10964893] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:31:37.187 INFO 4486 --- [host-check-agent-status3,a1adefa9ba276d36,32b10f1c10964893] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:31:37.187 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:31:37.190338726+08:00 2024-10-16 13:31:37.188 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:31:37.187+08:00 2024-10-16 13:31:37.188 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:31:37.183+08:00 and status AVAILABLE 2024-10-16 13:31:37.188 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:31:37.183+08:00 and status AVAILABLE 2024-10-16 13:31:37.188 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:31:37.187+08:00 2024-10-16 13:31:37.189 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:37.189 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.189 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:31:37.190 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:31:37.190 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:31:37.191 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:31:37.187+08:00 and status AVAILABLE 2024-10-16 13:31:37.191 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.191 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:31:37.191 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:31:37.187+08:00 and status AVAILABLE 2024-10-16 13:31:37.192 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:31:37.193 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:31:37.193 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:31:37.193 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.194 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.195 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.196 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:31:37.197 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 13:31:37.198 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.200 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:31:37.201 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:31:37.201 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:31:37.202 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:31:37.202 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:37.202 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:31:37.202 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:31:37.203 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:37.204 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:31:37.207 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.208 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:31:37.209 INFO 4486 --- [host-check-agent-status4,a1adefa9ba276d36,be8db65369c241b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:31:37.211 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.211 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.211 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.211 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.211 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.212 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.212 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.213 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:31:37.214 INFO 4486 --- [host-check-agent-status4,a1adefa9ba276d36,be8db65369c241b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:31:37.214 INFO 4486 --- [host-check-agent-status4,a1adefa9ba276d36,be8db65369c241b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:31:37.214 INFO 4486 --- [host-check-agent-status4,a1adefa9ba276d36,be8db65369c241b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:31:37.220 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.223 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 13:31:37.223 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:37.224 INFO 4486 --- [ocp-schedules-28,66f966b389f70994,7b7cbb301c01e64f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:31:37.226 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:37.227 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.227 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:37.231 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:31:37.230 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:31:37.233 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 13:31:37.239 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.240 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.241 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:31:37.243 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 13:31:37.256 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.256 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:31:37.257 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:31:37.262 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.263 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:31:37.265 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:31:37.267 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:31:37.288 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.289 INFO 4486 --- [ocp-schedules-15,f1ecd04a37979687,64a5603c1e451925] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:31:37.291 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:31:37.292 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:31:37.293 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:31:37.297 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:31:37.299 WARN 4486 --- [scanner-13244627-1,66f966b389f70994,be3168ae3a942790] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:31:37.304 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:31:37.319 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:37.319 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:37.321 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:37.335 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:31:37.351 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:31:37.359 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:31:37.363 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:31:37.364 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:31:37.371 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:31:37.376 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:31:37.387 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:37.393 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:37.395 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:37.396 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:31:37.412 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:31:37.416 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:31:37.425 INFO 4486 --- [host-check-host-status3,f2fee5a76bd4cae0,2acaf5ce973a8479] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:31:37.426 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:31:37.428 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:31:37.437 INFO 4486 --- [host-check-host-status4,f2fee5a76bd4cae0,490302494410caa2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:31:37.441 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:31:37.452 INFO 4486 --- [host-check-host-status2,f2fee5a76bd4cae0,d56886a5c7a32597] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:31:37.487 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:31:37.499 INFO 4486 --- [host-check-host-status1,f2fee5a76bd4cae0,d264657bc42f9baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:31:37.517 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:37.519 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.519 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:31:37.521 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.521 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.522 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:31:37.532 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.532 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.532 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.532 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.532 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.533 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.561 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:31:37.601 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:31:37.644 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:37.658 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:37.661 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.662 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:31:37.663 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:31:37.664 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.664 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:31:37.665 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.666 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.667 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:31:37.676 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.676 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.676 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.676 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.676 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.676 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.684 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:31:37.894 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:31:37.945 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:37.947 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.948 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:31:37.949 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.950 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.951 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:31:37.960 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.960 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.960 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.960 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.960 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:37.961 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:31:37.978 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:31:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:31:37.994 INFO 4486 --- [prometheus-minute-schedule-1,3efa08ca4caa5605,1e1508450ddd91ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:37.995 INFO 4486 --- [ocp-schedules-10,cb420a2d414b4eff,929dfebe64f8f277] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 13:31:37.995 INFO 4486 --- [ocp-schedules-10,cb420a2d414b4eff,929dfebe64f8f277] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 13:31:38.077 INFO 4486 --- [host-check-clockdiff3,725aa09bb1cfe70a,a6725dc02b130589] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056698 3 3 , error= 2024-10-16 13:31:38.079 INFO 4486 --- [host-check-clockdiff4,725aa09bb1cfe70a,3aca68a6ab9c6046] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056698 3 3 , error= 2024-10-16 13:31:38.080 INFO 4486 --- [prometheus-minute-schedule-1,3efa08ca4caa5605,1e1508450ddd91ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:38.110 INFO 4486 --- [pool-10-thread-10,730b38682ef3c58f,aaeaf904ac81435b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 13:31:38.111 INFO 4486 --- [pool-10-thread-10,730b38682ef3c58f,aaeaf904ac81435b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 13:31:38.116 INFO 4486 --- [pool-10-thread-11,730b38682ef3c58f,28375d0cfeeb7af9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 13:31:38.117 INFO 4486 --- [pool-10-thread-9,730b38682ef3c58f,cb444faf1b8859ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 13:31:38.118 INFO 4486 --- [pool-10-thread-12,730b38682ef3c58f,0d45828da73d921b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 13:31:38.121 INFO 4486 --- [ForkJoinPool-6-worker-10,730b38682ef3c58f,e231bb370ecc104e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 13:31:38.121 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 13:31:38.121 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 13:31:38.121 INFO 4486 --- [ForkJoinPool-6-worker-10,730b38682ef3c58f,e231bb370ecc104e] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 13:31:38.121 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 13:31:38.121 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 13:31:38.121 INFO 4486 --- [pool-10-thread-9,730b38682ef3c58f,cb444faf1b8859ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 13:31:38.121 INFO 4486 --- [pool-10-thread-9,730b38682ef3c58f,cb444faf1b8859ac] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 13:31:38.127 INFO 4486 --- [pool-overview-container-1,94a7aa5665bcbd57,ed34a9e5414175bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.130 INFO 4486 --- [pool-overview-container-1,94a7aa5665bcbd57,ed34a9e5414175bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:31:38.137 INFO 4486 --- [pool-10-thread-10,730b38682ef3c58f,aaeaf904ac81435b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 13:31:38.145 INFO 4486 --- [pool-10-thread-10,730b38682ef3c58f,aaeaf904ac81435b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 13:31:38.149 INFO 4486 --- [ForkJoinPool-6-worker-10,730b38682ef3c58f,e231bb370ecc104e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.150 INFO 4486 --- [ForkJoinPool-6-worker-10,730b38682ef3c58f,e231bb370ecc104e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:31:38.151 INFO 4486 --- [ForkJoinPool-6-worker-10,730b38682ef3c58f,e231bb370ecc104e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.152 INFO 4486 --- [ForkJoinPool-6-worker-10,730b38682ef3c58f,e231bb370ecc104e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:31:38.153 INFO 4486 --- [ForkJoinPool-6-worker-5,730b38682ef3c58f,ca9e3fecb24f0435] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.156 INFO 4486 --- [ForkJoinPool-6-worker-5,730b38682ef3c58f,ca9e3fecb24f0435] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:31:38.158 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.159 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:31:38.160 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.162 INFO 4486 --- [pool-overview-container-1,94a7aa5665bcbd57,ed34a9e5414175bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.165 INFO 4486 --- [pool-overview-container-1,94a7aa5665bcbd57,ed34a9e5414175bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:31:38.166 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.167 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.167 WARN 4486 --- [ForkJoinPool-6-worker-10,730b38682ef3c58f,e231bb370ecc104e] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:31:38.167 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:31:38.168 INFO 4486 --- [ForkJoinPool-6-worker-5,730b38682ef3c58f,ca9e3fecb24f0435] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 13:31:38.168 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:31:38.168 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:31:38.168 INFO 4486 --- [ForkJoinPool-6-worker-5,730b38682ef3c58f,ca9e3fecb24f0435] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.170 INFO 4486 --- [ForkJoinPool-6-worker-5,730b38682ef3c58f,ca9e3fecb24f0435] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:31:38.171 INFO 4486 --- [ForkJoinPool-6-worker-5,730b38682ef3c58f,ca9e3fecb24f0435] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.172 INFO 4486 --- [ForkJoinPool-6-worker-5,730b38682ef3c58f,ca9e3fecb24f0435] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:31:38.172 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.173 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:31:38.177 INFO 4486 --- [ForkJoinPool-6-worker-7,730b38682ef3c58f,165f585acf1cb3fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.178 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.180 INFO 4486 --- [ForkJoinPool-6-worker-7,730b38682ef3c58f,165f585acf1cb3fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:31:38.184 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:31:38.187 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:31:38.189 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.191 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:31:38.195 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.197 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:31:38.199 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.200 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:31:38.206 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.207 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:31:38.212 INFO 4486 --- [ForkJoinPool-6-worker-7,730b38682ef3c58f,165f585acf1cb3fb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:31:38.213 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:38.214 INFO 4486 --- [ForkJoinPool-6-worker-7,730b38682ef3c58f,165f585acf1cb3fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.215 INFO 4486 --- [ForkJoinPool-6-worker-7,730b38682ef3c58f,165f585acf1cb3fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:31:38.217 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.220 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.221 INFO 4486 --- [ForkJoinPool-6-worker-7,730b38682ef3c58f,165f585acf1cb3fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.221 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:31:38.222 INFO 4486 --- [ForkJoinPool-6-worker-7,730b38682ef3c58f,165f585acf1cb3fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:31:38.222 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:31:38.225 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.226 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.227 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:31:38.231 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:31:38.235 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.240 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.240 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:31:38.241 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.241 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:31:38.246 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:31:38.251 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:31:38.253 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.253 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:31:38.254 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.255 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:31:38.257 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 13:31:38.257 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.259 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:31:38.265 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.266 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:31:38.272 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.273 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:31:38.273 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:31:38.276 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.276 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:31:38.281 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 13:31:38.281 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.284 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:31:38, 2024-10-16 13:31:38] 2024-10-16 13:31:38.285 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 13:31:38.291 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.292 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.292 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:38.294 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:31:38.297 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:31:38.299 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.303 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:31:38.304 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.306 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:31:38.306 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:31:38, 2024-10-16 13:31:38] 2024-10-16 13:31:38.311 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:31:38.312 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 13:31:38.313 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.314 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.315 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:31:38, 2024-10-16 13:31:38] 2024-10-16 13:31:38.316 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:31:38.318 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.320 INFO 4486 --- [ForkJoinPool-6-worker-12,730b38682ef3c58f,361252950927e0e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:31:38.323 INFO 4486 --- [pool-10-thread-12,730b38682ef3c58f,0d45828da73d921b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 13:31:38.330 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:38.330 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:31:38.332 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:38.332 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.332 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.334 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:31:38, 2024-10-16 13:31:38] 2024-10-16 13:31:38.334 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.334 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:31:38.335 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:38.336 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.337 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:31:38.340 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 13:31:38.340 INFO 4486 --- [ForkJoinPool-6-worker-1,730b38682ef3c58f,39005d25403ad597] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 13:31:38.341 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 13:31:38.341 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:38.342 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.346 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 13:31:38.347 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:38.348 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:38.348 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:38.348 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:38.348 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:31:38.348 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.348 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.348 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:31:38.350 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:31:38, 2024-10-16 13:31:38] 2024-10-16 13:31:38.378 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:31:38.382 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.383 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:38.384 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:31:38, 2024-10-16 13:31:38] 2024-10-16 13:31:38.385 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:38.388 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 13:31:38.388 INFO 4486 --- [ForkJoinPool-6-worker-3,730b38682ef3c58f,8e49a2f70af03ec1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 13:31:38.391 INFO 4486 --- [pool-10-thread-11,730b38682ef3c58f,28375d0cfeeb7af9] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 13:31:38.393 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:31:38.421 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:31:38.465 INFO 4486 --- [ocp-schedules-17,44f922686e009c3b,60636365eb3a362b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:31:38.465 INFO 4486 --- [ocp-schedules-17,44f922686e009c3b,60636365eb3a362b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:31:38.616 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:31:38.424598, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:31:38.616 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:31:38.424598, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:31:38.658 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:31:38.424598} } 2024-10-16 13:31:38.713 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:31:38.719 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:31:38.722 INFO 4486 --- [ocp-schedules-9,c0075e8cf8d36344,139f91d7e988a484] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:31:38.827 INFO 4486 --- [ocp-schedules-29,622f45e62b08eaad,d14d522701544da2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:31:39.030 INFO 4486 --- [ocp-async-4,4989b127d2bcb7ea,9ef3f28d8ff13295] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:39.204 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,b1581def1f3582af,4bac51b07d24feec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b1581def1f3582af, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:39.204 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,b1581def1f3582af,4bac51b07d24feec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b1581def1f3582af, duration=1 ms] 2024-10-16 13:31:39.707 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:40.006 INFO 4486 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:31:40.006 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:31:40.024 INFO 4486 --- [Check obproxy2,e08d5571f346f68e,f0020267cceb902e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:31:40.024 INFO 4486 --- [Check obproxy2,e08d5571f346f68e,f0020267cceb902e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:40.025 INFO 4486 --- [Check obproxy1,0c1b92d8b03af698,545fb8ccbf76d0e3] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:31:40.025 INFO 4486 --- [Check obproxy1,0c1b92d8b03af698,545fb8ccbf76d0e3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:40.029 INFO 4486 --- [Collect obproxy1,62bf376d5ef4000a,ca65fb20c5c6ed4a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:31:40.029 INFO 4486 --- [Collect obproxy1,62bf376d5ef4000a,ca65fb20c5c6ed4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:40.030 INFO 4486 --- [Collect obproxy2,28f292a5954dd280,31604f8348a1851a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:31:40.030 INFO 4486 --- [Collect obproxy2,28f292a5954dd280,31604f8348a1851a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:31:40.049 INFO 4486 --- [Check obproxy2,e08d5571f346f68e,f0020267cceb902e] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:31:40.049 INFO 4486 --- [Check obproxy2,e08d5571f346f68e,f0020267cceb902e] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:31:40.051 INFO 4486 --- [Collect obproxy2,28f292a5954dd280,31604f8348a1851a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:31:40.053 INFO 4486 --- [Check obproxy1,0c1b92d8b03af698,545fb8ccbf76d0e3] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:31:40.054 INFO 4486 --- [Check obproxy1,0c1b92d8b03af698,545fb8ccbf76d0e3] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:31:40.057 INFO 4486 --- [Collect obproxy2,28f292a5954dd280,31604f8348a1851a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:31:40.062 INFO 4486 --- [Collect obproxy2,28f292a5954dd280,31604f8348a1851a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:31:40.062 INFO 4486 --- [Collect obproxy1,62bf376d5ef4000a,ca65fb20c5c6ed4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:31:40.067 INFO 4486 --- [Collect obproxy1,62bf376d5ef4000a,ca65fb20c5c6ed4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:31:40.072 INFO 4486 --- [Collect obproxy1,62bf376d5ef4000a,ca65fb20c5c6ed4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:31:40.227 INFO 4486 --- [ocp-schedules-12,38082e7eabc35739,3d80c61bed304df1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:31:40.789 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d91a874c739e206b,d31f374b8583ae08] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d91a874c739e206b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:40.790 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d91a874c739e206b,d31f374b8583ae08] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d91a874c739e206b, duration=2 ms] 2024-10-16 13:31:40.994 INFO 4486 --- [prometheus-minute-schedule-1,2936747c7cb44a80,6f7cba1be9bd3790] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:40.995 INFO 4486 --- [prometheus-hour-schedule-1,d35501f9267a10a6,ab6106a1f0dab8cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:31:41.000 INFO 4486 --- [prometheus-minute-schedule-1,2936747c7cb44a80,6f7cba1be9bd3790] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:41.030 INFO 4486 --- [ocp-async-10,6741c6bcd0df81f9,e286620b79024879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:41.173 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,89979f1c85e47ec4,99635d707b117cc7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=89979f1c85e47ec4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:41.173 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,89979f1c85e47ec4,99635d707b117cc7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=89979f1c85e47ec4, duration=1 ms] 2024-10-16 13:31:42.077 INFO 4486 --- [ocp-schedules-25,446950d400f13339,e0aaec0c4bbdeb34] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:31:42.708 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:42.984 INFO 4486 --- [ocp-async-8,65902e41b8fa6769,24af9ad83d38c567] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:31:43.030 INFO 4486 --- [ocp-async-6,d7d7e96e5a0a6887,694b717d0d06405b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:43.994 INFO 4486 --- [prometheus-minute-schedule-1,21c5ba2833751e89,7a2dc586d71c359b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:44.003 INFO 4486 --- [prometheus-minute-schedule-1,21c5ba2833751e89,7a2dc586d71c359b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:45.030 INFO 4486 --- [ocp-async-2,adeb5daab87beebf,aea96656ae66b794] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:45.431 INFO 4486 --- [ocp-schedules-10,62eb0d105c0a0180,3974794b170c712b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:31:45.436 INFO 4486 --- [ocp-schedules-10,62eb0d105c0a0180,3974794b170c712b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:31:45.436 INFO 4486 --- [ocp-schedules-10,62eb0d105c0a0180,3974794b170c712b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:31:45.447 INFO 4486 --- [ocp-schedules-9,fabf8636ad02d4c4,7b30d3903d4b0293] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:31:45.447 INFO 4486 --- [ocp-schedules-9,fabf8636ad02d4c4,7b30d3903d4b0293] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:31:45.709 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:46.986 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,067e72d65ab0e200,ba2088121094ad78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=067e72d65ab0e200, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:46.986 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,067e72d65ab0e200,ba2088121094ad78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=067e72d65ab0e200, duration=2 ms] 2024-10-16 13:31:46.994 INFO 4486 --- [prometheus-minute-schedule-1,49fe329954b7104f,22880d4c37330d17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:47.003 INFO 4486 --- [prometheus-minute-schedule-1,49fe329954b7104f,22880d4c37330d17] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:47.031 INFO 4486 --- [ocp-async-9,a9844a5d516330f2,ae23b281b49d8bb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:47.074 INFO 4486 --- [ocp-schedules-18,2057c636234bb05e,4cf58874aa36ffd0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:31:47.076 INFO 4486 --- [ocp-schedules-6,0c3670bb14f96e57,421c5cc2f08d7d92] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:31:47.189 INFO 4486 --- [ocp-schedules-20,f2a67ec18c578f1b,09bd5f268ef7a305] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:31:47.246 INFO 4486 --- [ocp-schedules-30,a4e4eb383e6cb8bc,8e9328d3584cf82a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 13:31:47.421 INFO 4486 --- [ocp-schedules-27,73e912aa91d0be44,43b50e2ceda936e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:31:47.464 WARN 4486 --- [scanner-13244857-1,73e912aa91d0be44,aa54d4d39ebcf91a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:31:47.469 INFO 4486 --- [alarm-detect-4,73e912aa91d0be44,ebfa9007d3160d81] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:47.473 INFO 4486 --- [alarm-detect-4,73e912aa91d0be44,ebfa9007d3160d81] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:47.476 INFO 4486 --- [alarm-detect-4,73e912aa91d0be44,ebfa9007d3160d81] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:31:47.486 INFO 4486 --- [alarm-detect-17,73e912aa91d0be44,0cdea60b054e8bd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:47.489 INFO 4486 --- [alarm-detect-5,73e912aa91d0be44,6fd216972121635e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:47.490 INFO 4486 --- [alarm-detect-17,73e912aa91d0be44,0cdea60b054e8bd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:47.493 INFO 4486 --- [alarm-detect-5,73e912aa91d0be44,6fd216972121635e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:47.495 INFO 4486 --- [alarm-detect-17,73e912aa91d0be44,0cdea60b054e8bd3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:31:47.498 INFO 4486 --- [alarm-detect-5,73e912aa91d0be44,6fd216972121635e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:31:47.529 INFO 4486 --- [alarm-detect-18,73e912aa91d0be44,125ebd66c7fc3d23] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056675 ~ 1729056697 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056690, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056690, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056690=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:31:47.540 INFO 4486 --- [alarm-detect-18,73e912aa91d0be44,125ebd66c7fc3d23] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:31:47.561 INFO 4486 --- [alarm-detect-16,73e912aa91d0be44,e1510f412db3e4c3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:47.565 INFO 4486 --- [alarm-detect-16,73e912aa91d0be44,e1510f412db3e4c3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:47.570 INFO 4486 --- [alarm-detect-16,73e912aa91d0be44,e1510f412db3e4c3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:31:48.710 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:48.758 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:31:49.031 INFO 4486 --- [ocp-async-6,fb3c260bccbb222c,2fdfa1c59e01cf08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:49.212 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,b623aa4a41a0e7e6,1483ebe83daa1987] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b623aa4a41a0e7e6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:49.212 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,b623aa4a41a0e7e6,1483ebe83daa1987] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b623aa4a41a0e7e6, duration=1 ms] 2024-10-16 13:31:49.794 INFO 4486 --- [ocp-schedules-24,a5b29c66771a6fca,057b17d7872cd80e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:31:49.795 INFO 4486 --- [ocp-schedules-24,a5b29c66771a6fca,057b17d7872cd80e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:31:49.795 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:31:49.795 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:31:49.795 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:31:49.795 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:31:49.795 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:31:49.800 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:31:49.801 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:31:49.801 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:31:49.801 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:49.802 INFO 4486 --- [ocp-schedules-24,a5b29c66771a6fca,057b17d7872cd80e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:31:49.802 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:31:49.802 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:31:49.802 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:31:49.802 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:31:49.802 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:49.803 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:31:49.803 INFO 4486 --- [ocp-schedules-24,a5b29c66771a6fca,057b17d7872cd80e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:31:49.825 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:31:49.825 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:31:49.827 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:31:49.831 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:31:49.834 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:31:49.834 INFO 4486 --- [ocp-schedules-24,a5b29c66771a6fca,057b17d7872cd80e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:31:49.839 INFO 4486 --- [ocp-schedules-24,a5b29c66771a6fca,057b17d7872cd80e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:31:49.994 INFO 4486 --- [prometheus-minute-schedule-1,3edc7cff75b21909,27d5bd3d0731e94f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:50.001 INFO 4486 --- [prometheus-minute-schedule-1,3edc7cff75b21909,27d5bd3d0731e94f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:50.234 INFO 4486 --- [ocp-schedules-28,40fdc2845b94bafc,228eeb85dccce997] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:31:50.356 INFO 4486 --- [ocp-schedules-3,321a6d6945809c41,424cf279e6ca8163] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:31:50.356 INFO 4486 --- [ocp-schedules-3,321a6d6945809c41,424cf279e6ca8163] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:31:50.357 INFO 4486 --- [ocp-schedules-3,321a6d6945809c41,424cf279e6ca8163] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:31:50.357 INFO 4486 --- [ocp-schedules-3,321a6d6945809c41,424cf279e6ca8163] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:31:50.358 INFO 4486 --- [ocp-schedules-3,321a6d6945809c41,424cf279e6ca8163] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:31:50.358 INFO 4486 --- [ocp-schedules-3,321a6d6945809c41,424cf279e6ca8163] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:31:50.794 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3282dab932577a2b,62dcf05225970acc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3282dab932577a2b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:50.794 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3282dab932577a2b,62dcf05225970acc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3282dab932577a2b, duration=1 ms] 2024-10-16 13:31:51.031 INFO 4486 --- [ocp-async-2,b5058f4ff0522f5c,a610b0c3312c9c70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:51.177 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b9af25a02b4ff90b,7c41002cbf56844e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b9af25a02b4ff90b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:51.177 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b9af25a02b4ff90b,7c41002cbf56844e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b9af25a02b4ff90b, duration=1 ms] 2024-10-16 13:31:51.710 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:52.113 INFO 4486 --- [ocp-schedules-7,2bbccc54a268410d,2529c959e90eca59] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:31:52.985 INFO 4486 --- [ocp-async-5,e5a3d68699faa94f,911367668565532c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:31:52.994 INFO 4486 --- [prometheus-minute-schedule-1,aab4a569f0bf3894,89d9580ac0e534a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:53.000 INFO 4486 --- [prometheus-minute-schedule-1,aab4a569f0bf3894,89d9580ac0e534a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:53.031 INFO 4486 --- [ocp-async-9,2e897eb8633f82ed,7b2485e7a26b75ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:54.711 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:55.031 INFO 4486 --- [ocp-async-8,3896e38c1261d1fa,ea10f5ac5995dcba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:55.994 INFO 4486 --- [prometheus-minute-schedule-1,ffa78fd09d1077ec,ae63669f9b103348] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:56.000 INFO 4486 --- [prometheus-minute-schedule-1,ffa78fd09d1077ec,ae63669f9b103348] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:56.991 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5bdcbdf43c444c53,06aad8f49ad1c623] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5bdcbdf43c444c53, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:56.992 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5bdcbdf43c444c53,06aad8f49ad1c623] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5bdcbdf43c444c53, duration=2 ms] 2024-10-16 13:31:57.032 INFO 4486 --- [ocp-async-4,9c56d32a89137ae2,c2db875fa03d656f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:57.601 INFO 4486 --- [ocp-schedules-3,936a3fcff8292572,e4da899c314e9ae9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:31:57.621 WARN 4486 --- [scanner-13245157-1,936a3fcff8292572,ef75e1b36938bc69] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:31:57.662 INFO 4486 --- [alarm-detect-1,936a3fcff8292572,7132ca434c6f43b2] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056690 ~ 1729056707 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056705, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056705, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056705=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:31:57.712 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:31:58.326 INFO 4486 --- [ocp-schedules-16,7106e1401f1e7071,af7862705a830755] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:31:58.326 INFO 4486 --- [ocp-schedules-16,d9dd51771f920a57,bb07a3ad2db33f60] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:31:58.329 INFO 4486 --- [ocp-schedules-16,d9dd51771f920a57,bb07a3ad2db33f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:31:58.356 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.357 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:58.378 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.384 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:58.389 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:58.390 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.390 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:31:58.400 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.405 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:58.418 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:58.418 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.419 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:58.420 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:31:58.420 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.423 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:31:58.425 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:58.435 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.435 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:58.446 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.457 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:58.458 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.459 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:58.459 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:31:58.464 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:31:58.468 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:31:58.470 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:31:58.498 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:31:58.534 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:58.536 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.537 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:31:58.559 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:31:58.559 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:31:58.579 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:31:58.588 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:58.589 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.590 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:31:58.590 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:58.592 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.593 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:31:58.598 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.599 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:31:58.599 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:31:58.602 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:31:58.605 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:58.630 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.630 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:58.658 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.658 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:58.682 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.682 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:58.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:31:58.737 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.737 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:31:58.767 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:31:58.794 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:58.796 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:58.797 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:31:58.801 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:31:58.806 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:31:58.811 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:31:58.813 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:31:58.833 INFO 4486 --- [ocp-schedules-30,49c330bc0b12b777,7c351689f9aecb30] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:31:58.837 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:31:58.933 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:31:58.933 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:31:58.941 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:31:58.941 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:31:58.941 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:31:58.942 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:31:58.951 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:31:58.951 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:31:58.951 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:31:58.951 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:31:58.993 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:31:58.994 INFO 4486 --- [prometheus-minute-schedule-1,a946b14a017a20fa,aa38adef9a655271] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056720, interval=60 2024-10-16 13:31:59.000 INFO 4486 --- [prometheus-minute-schedule-1,a946b14a017a20fa,aa38adef9a655271] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:31:59.008 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:31:59.022 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:31:59.022 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:31:59.022 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:31:59.022 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:31:59.022 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:31:59.030 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:31:59.030 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:31:59.031 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:31:59.032 INFO 4486 --- [ocp-async-10,8a8e7b3e3f58bb69,32d098a7c82d5979] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:31:59.032 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:31:59.064 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:31:59.064 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:31:59.216 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ad8b25581c88808f,b9624bcbcaeed840] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ad8b25581c88808f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:31:59.216 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ad8b25581c88808f,b9624bcbcaeed840] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ad8b25581c88808f, duration=1 ms] 2024-10-16 13:31:59.528 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:31:59.993 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.042000049S 2024-10-16 13:31:59.993 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:32:00.062 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:32:00.062 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:32:00.086 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:32:00.162 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:32:00.162 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:32:00.162 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:32:00.162 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:32:00.162 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:32:00.162 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:32:00.171 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:32:00.240 INFO 4486 --- [ocp-schedules-6,4d6fb0c2f5bdfd83,f06c9eb5a26287f8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:32:00.302 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] o.s.boot.SpringApplication : Started application in 1.361 seconds (JVM running for 77623.354) 2024-10-16 13:32:00.303 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.361734098S 2024-10-16 13:32:00.699 INFO 4486 --- [ocp-schedules-24,dc53e5a374a7465e,8d7718acb119190f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:32:00.713 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:00.800 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c17ce883aa01fa2b,460fee8536dcc09c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c17ce883aa01fa2b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:00.800 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c17ce883aa01fa2b,460fee8536dcc09c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c17ce883aa01fa2b, duration=2 ms] 2024-10-16 13:32:00.971 INFO 4486 --- [ocp-schedules-21,65c4a7fa80a640cc,df68c59f33fb295b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:32:00.971+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:32:00.985 INFO 4486 --- [ocp-schedules-21,65c4a7fa80a640cc,df68c59f33fb295b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:32:00.996 INFO 4486 --- [ocp-schedules-21,65c4a7fa80a640cc,df68c59f33fb295b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:32:01.008 INFO 4486 --- [ocp-schedules-21,65c4a7fa80a640cc,df68c59f33fb295b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:32:01.032 INFO 4486 --- [ocp-async-6,ea0cde223cc2794d,70b7f54ded927046] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:01.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,8ce1c8ab321bc2ac,7c9b64f141ac24cc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8ce1c8ab321bc2ac, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:01.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,8ce1c8ab321bc2ac,7c9b64f141ac24cc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8ce1c8ab321bc2ac, duration=1 ms] 2024-10-16 13:32:01.994 INFO 4486 --- [prometheus-minute-schedule-1,d52d75acfed9a82d,29efb46aa0a8f03a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:02.000 INFO 4486 --- [prometheus-minute-schedule-1,d52d75acfed9a82d,29efb46aa0a8f03a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:02.985 INFO 4486 --- [ocp-async-8,a23ade24d003a619,5bb948a1c6db8f38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:32:03.032 INFO 4486 --- [ocp-async-7,bd7a49273f7712a7,c8f43bd50dd41132] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:03.449 INFO 4486 --- [ocp-schedules-22,3e9a1c5c1008e45c,3573aeaeff7ef166] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:32:03.450 INFO 4486 --- [ocp-schedules-22,3e9a1c5c1008e45c,3573aeaeff7ef166] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:03.451 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:32:03.451 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:32:03.451 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:32:03.713 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:04.994 INFO 4486 --- [prometheus-minute-schedule-1,2f7cdedb46e77eee,87e5093ee8f53f0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:05.005 INFO 4486 --- [prometheus-minute-schedule-1,2f7cdedb46e77eee,87e5093ee8f53f0c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:05.024 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1955f7fcdd8dc87c,abc0f044a3a92cd2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=1955f7fcdd8dc87c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:32:05.024 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1955f7fcdd8dc87c,abc0f044a3a92cd2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=1955f7fcdd8dc87c, duration=3 ms] 2024-10-16 13:32:05.033 INFO 4486 --- [ocp-async-1,7f4360eb0acd87d9,61106a8bcb55aa19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:06.715 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:06.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,4f6e2160c80c70aa,ad5de6b442f77e90] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4f6e2160c80c70aa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:06.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,4f6e2160c80c70aa,ad5de6b442f77e90] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4f6e2160c80c70aa, duration=1 ms] 2024-10-16 13:32:07.033 INFO 4486 --- [ocp-async-9,1bfdeafc2b505672,7b49da33d7445496] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:07.066 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:32:07.067 INFO 4486 --- [ocp-schedules-12,8681667f5126cf84,8a434879ad587a2f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:32:07.068 INFO 4486 --- [ocp-schedules-12,8681667f5126cf84,8a434879ad587a2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:07.081 INFO 4486 --- [ocp-schedules-15,98615afcde47b1b8,5e302841e2cc32bd] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:32:07.084 INFO 4486 --- [ocp-schedules-15,98615afcde47b1b8,5e302841e2cc32bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:07.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:32:07.089 INFO 4486 --- [ocp-schedules-25,145b38c9af2e4a9c,068dd55a5ffb9211] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:32:07.089 INFO 4486 --- [ocp-schedules-1,dddad9498db420f5,379d465150c7f765] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:32:07.090 INFO 4486 --- [ocp-schedules-17,580fa9a0d2d0e1d1,1fa8620cbeed590f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:32:07.090 INFO 4486 --- [host-check-clockdiff1,dddad9498db420f5,1625b262d27a264e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:32:07.090 INFO 4486 --- [host-check-agent-status1,145b38c9af2e4a9c,e77e2daa7cca3612] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:32:07.090 INFO 4486 --- [host-check-agent-status2,145b38c9af2e4a9c,1773c77ea285e1f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:32:07.091 INFO 4486 --- [host-check-agent-status4,145b38c9af2e4a9c,d1b4564e7792b348] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:32:07.091 INFO 4486 --- [host-check-agent-status3,145b38c9af2e4a9c,15fd0e58858c7bea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:32:07.092 INFO 4486 --- [host-check-clockdiff2,dddad9498db420f5,2a46ffadae11a0ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:32:07.092 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:32:07.094 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:32:07.094 INFO 4486 --- [host-check-clockdiff3,dddad9498db420f5,787df5de1e0733bb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:32:07.096 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:32:07.096 INFO 4486 --- [host-check-clockdiff4,dddad9498db420f5,c2dc088b766a6673] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:32:07.097 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:32:07.115 INFO 4486 --- [host-check-agent-status3,145b38c9af2e4a9c,15fd0e58858c7bea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:07.115 INFO 4486 --- [host-check-agent-status2,145b38c9af2e4a9c,1773c77ea285e1f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:07.133 INFO 4486 --- [host-check-agent-status1,145b38c9af2e4a9c,e77e2daa7cca3612] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:07.133 INFO 4486 --- [host-check-agent-status4,145b38c9af2e4a9c,d1b4564e7792b348] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:07.133 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:07.134 INFO 4486 --- [host-check-clockdiff4,dddad9498db420f5,c2dc088b766a6673] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056727 3 3 , error= 2024-10-16 13:32:07.136 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:07.136 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:07.139 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-agent-status3,145b38c9af2e4a9c,15fd0e58858c7bea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-agent-status2,145b38c9af2e4a9c,1773c77ea285e1f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-agent-status1,145b38c9af2e4a9c,e77e2daa7cca3612] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-agent-status4,145b38c9af2e4a9c,d1b4564e7792b348] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:07.144 INFO 4486 --- [host-check-clockdiff1,dddad9498db420f5,1625b262d27a264e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056727 0 0 , error= 2024-10-16 13:32:07.149 INFO 4486 --- [host-check-clockdiff3,dddad9498db420f5,787df5de1e0733bb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056727 3 3 , error= 2024-10-16 13:32:07.149 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.150 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:07.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:32:07.151 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:32:07.173 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.174 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:32:07.175 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:32:07.175 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:32:07.176 INFO 4486 --- [host-check-agent-status3,145b38c9af2e4a9c,15fd0e58858c7bea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:32:07.176 INFO 4486 --- [host-check-agent-status2,145b38c9af2e4a9c,1773c77ea285e1f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:32:07.178 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:32:07.180853703+08:00 2024-10-16 13:32:07.178 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:32:07.180320282+08:00 2024-10-16 13:32:07.179 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:32:07.178+08:00 2024-10-16 13:32:07.179 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:32:07.178+08:00 2024-10-16 13:32:07.179 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:32:07.179 INFO 4486 --- [host-check-agent-status1,145b38c9af2e4a9c,e77e2daa7cca3612] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:32:07.182 INFO 4486 --- [host-check-agent-status3,145b38c9af2e4a9c,15fd0e58858c7bea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:32:07.182 INFO 4486 --- [host-check-agent-status2,145b38c9af2e4a9c,1773c77ea285e1f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:32:07.182 INFO 4486 --- [host-check-agent-status2,145b38c9af2e4a9c,1773c77ea285e1f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:32:07.182 INFO 4486 --- [host-check-agent-status3,145b38c9af2e4a9c,15fd0e58858c7bea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:32:07.182 INFO 4486 --- [host-check-agent-status2,145b38c9af2e4a9c,1773c77ea285e1f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:32:07.182 INFO 4486 --- [host-check-agent-status3,145b38c9af2e4a9c,15fd0e58858c7bea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:32:07.183 INFO 4486 --- [host-check-agent-status4,145b38c9af2e4a9c,d1b4564e7792b348] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:32:07.185 INFO 4486 --- [host-check-agent-status1,145b38c9af2e4a9c,e77e2daa7cca3612] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:32:07.185 INFO 4486 --- [host-check-agent-status1,145b38c9af2e4a9c,e77e2daa7cca3612] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:32:07.185 INFO 4486 --- [host-check-agent-status1,145b38c9af2e4a9c,e77e2daa7cca3612] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:32:07.186 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:32:07.187 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:32:07.186772709+08:00 2024-10-16 13:32:07.187 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:32:07.187+08:00 2024-10-16 13:32:07.188 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.188 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:32:07.188 INFO 4486 --- [host-check-agent-status4,145b38c9af2e4a9c,d1b4564e7792b348] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:32:07.188 INFO 4486 --- [host-check-agent-status4,145b38c9af2e4a9c,d1b4564e7792b348] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:32:07.188 INFO 4486 --- [host-check-agent-status4,145b38c9af2e4a9c,d1b4564e7792b348] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:32:07.189 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:32:07.192280651+08:00 2024-10-16 13:32:07.189 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:32:07.178+08:00 and status AVAILABLE 2024-10-16 13:32:07.190 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:32:07.189+08:00 2024-10-16 13:32:07.191 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:32:07.178+08:00 and status AVAILABLE 2024-10-16 13:32:07.191 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:32:07.187+08:00 and status AVAILABLE 2024-10-16 13:32:07.192 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:32:07.189+08:00 and status AVAILABLE 2024-10-16 13:32:07.193 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:32:07.193 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:32:07.193 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:32:07.193 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:32:07.200 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:32:07.202 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:07.202 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:07.203 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:07.203 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:07.204 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:07.204 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.204 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:07.205 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:07.208 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:07.209 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:07.215 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.216 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:07.227 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:07.228 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:07.229 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:07.229 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.230 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:07.231 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:07.232 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.233 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:32:07.242 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:07.243 INFO 4486 --- [ocp-schedules-7,a9eb757db353a53d,db096effd1912e95] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:32:07.258 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:32:07.261 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:32:07.263 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:32:07.269 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:32:07.270 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:32:07.273 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:32:07.292 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:07.293 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:32:07.294 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:07.294 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:07.303 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:32:07.320 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:07.324 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:32:07.328 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:32:07.334 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:32:07.334 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:32:07.340 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:32:07.343 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:32:07.351 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:07.358 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:07.358 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:07.372 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:32:07.384 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:32:07.385 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:32:07.394 INFO 4486 --- [host-check-host-status3,580fa9a0d2d0e1d1,39616a40598f6693] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:32:07.394 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:32:07.397 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:32:07.400 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:07.404 INFO 4486 --- [host-check-host-status2,580fa9a0d2d0e1d1,9e06fdf92f84fac2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:32:07.407 INFO 4486 --- [host-check-host-status4,580fa9a0d2d0e1d1,72ea06840fa750aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:32:07.452 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:32:07.464 INFO 4486 --- [host-check-host-status1,580fa9a0d2d0e1d1,5e1b80c0061154c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:32:07.716 INFO 4486 --- [ocp-schedules-20,001823ae28d4f66f,fb6182029274809f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:32:07.756 WARN 4486 --- [scanner-13245412-1,001823ae28d4f66f,87b9884352ea0ab1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:32:07.785 INFO 4486 --- [alarm-detect-11,001823ae28d4f66f,475121af7c1e5884] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:07.793 INFO 4486 --- [alarm-detect-11,001823ae28d4f66f,475121af7c1e5884] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:07.799 INFO 4486 --- [alarm-detect-11,001823ae28d4f66f,475121af7c1e5884] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:07.809 INFO 4486 --- [alarm-detect-19,001823ae28d4f66f,30d26d99f64767ea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:07.810 INFO 4486 --- [alarm-detect-7,001823ae28d4f66f,8aa4fd3ccf75c0b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:07.815 INFO 4486 --- [alarm-detect-19,001823ae28d4f66f,30d26d99f64767ea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:07.817 INFO 4486 --- [alarm-detect-7,001823ae28d4f66f,8aa4fd3ccf75c0b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:07.820 INFO 4486 --- [alarm-detect-19,001823ae28d4f66f,30d26d99f64767ea] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:07.822 INFO 4486 --- [alarm-detect-7,001823ae28d4f66f,8aa4fd3ccf75c0b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:07.889 INFO 4486 --- [alarm-detect-5,001823ae28d4f66f,000f62dcc7855148] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:07.893 INFO 4486 --- [alarm-detect-5,001823ae28d4f66f,000f62dcc7855148] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:07.897 INFO 4486 --- [alarm-detect-5,001823ae28d4f66f,000f62dcc7855148] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:07.994 INFO 4486 --- [prometheus-minute-schedule-1,3edbfd5f58337f68,08f1d9ef564610b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:08.001 INFO 4486 --- [prometheus-minute-schedule-1,3edbfd5f58337f68,08f1d9ef564610b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:08.053 INFO 4486 --- [host-check-clockdiff2,dddad9498db420f5,2a46ffadae11a0ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056728 3 3 , error= 2024-10-16 13:32:09.033 INFO 4486 --- [ocp-async-2,412b47ac63b7ad8f,81fbe9c3700d9cfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:09.219 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dc2a882684d29170,b2f7a1010bca1546] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dc2a882684d29170, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:09.220 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dc2a882684d29170,b2f7a1010bca1546] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dc2a882684d29170, duration=1 ms] 2024-10-16 13:32:09.716 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:10.007 INFO 4486 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:32:10.007 INFO 4486 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:32:10.026 INFO 4486 --- [Check obproxy2,956f09c8687d0603,8a455d0626504d38] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:32:10.027 INFO 4486 --- [Check obproxy1,f64d2e931b4af265,a85ebcb81c80d1aa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:32:10.030 INFO 4486 --- [Check obproxy2,956f09c8687d0603,8a455d0626504d38] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:10.031 INFO 4486 --- [Check obproxy1,f64d2e931b4af265,a85ebcb81c80d1aa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:10.031 INFO 4486 --- [Collect obproxy1,eb1dae390bef0e3d,bfd4034d24308b91] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:32:10.032 INFO 4486 --- [Collect obproxy1,eb1dae390bef0e3d,bfd4034d24308b91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:10.033 INFO 4486 --- [Collect obproxy2,68c4d3fc947923af,fc1d45ddf2aae08b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:32:10.033 INFO 4486 --- [Collect obproxy2,68c4d3fc947923af,fc1d45ddf2aae08b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:10.054 INFO 4486 --- [Collect obproxy2,68c4d3fc947923af,fc1d45ddf2aae08b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:32:10.055 INFO 4486 --- [Check obproxy2,956f09c8687d0603,8a455d0626504d38] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:32:10.055 INFO 4486 --- [Check obproxy2,956f09c8687d0603,8a455d0626504d38] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:32:10.059 INFO 4486 --- [Collect obproxy2,68c4d3fc947923af,fc1d45ddf2aae08b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:32:10.062 INFO 4486 --- [Collect obproxy1,eb1dae390bef0e3d,bfd4034d24308b91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:32:10.065 INFO 4486 --- [Collect obproxy2,68c4d3fc947923af,fc1d45ddf2aae08b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:32:10.067 INFO 4486 --- [Collect obproxy1,eb1dae390bef0e3d,bfd4034d24308b91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:32:10.071 INFO 4486 --- [Check obproxy1,f64d2e931b4af265,a85ebcb81c80d1aa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:32:10.071 INFO 4486 --- [Check obproxy1,f64d2e931b4af265,a85ebcb81c80d1aa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:32:10.073 INFO 4486 --- [Collect obproxy1,eb1dae390bef0e3d,bfd4034d24308b91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:32:10.247 INFO 4486 --- [ocp-schedules-6,28ea74bb98a46943,7544e2ae4099f4cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:32:10.814 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,80adb5e058db063a,32b6e49e9092e254] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=80adb5e058db063a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:10.814 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,80adb5e058db063a,32b6e49e9092e254] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=80adb5e058db063a, duration=1 ms] 2024-10-16 13:32:10.978 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:32:10.994 INFO 4486 --- [prometheus-minute-schedule-1,9bd31d9a64f4a239,33681330d9efc2f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:11.001 INFO 4486 --- [prometheus-minute-schedule-1,9bd31d9a64f4a239,33681330d9efc2f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:32:11.033 INFO 4486 --- [ocp-async-4,8d6532c6fcdec51b,3dd61fb69ae659f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:11.190 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ef6c9fb06b9ab9b5,293599b94046cd1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ef6c9fb06b9ab9b5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:11.190 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ef6c9fb06b9ab9b5,293599b94046cd1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ef6c9fb06b9ab9b5, duration=1 ms] 2024-10-16 13:32:12.717 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:12.985 INFO 4486 --- [ocp-async-1,aee4ccdae4f20a7e,0716a29af3744124] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:32:13.033 INFO 4486 --- [ocp-async-9,9ed2db18940fe6d0,577b965ef2d9c366] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:13.063 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056660, nodeCount=1000, affectRows=1000 2024-10-16 13:32:13.067 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056660, nodeCount=999, validCount=999 2024-10-16 13:32:13.068 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056660, nodeCount=1000, affectRows=1000 2024-10-16 13:32:13.069 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056660, nodeCount=1000, affectRows=1000 2024-10-16 13:32:13.071 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056660, nodeCount=999, validCount=999 2024-10-16 13:32:13.072 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056660, nodeCount=1000, validCount=1000 2024-10-16 13:32:13.076 ERROR 4486 --- [metric-archive-second-31,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990814) Duplicate entry '586774769288417280-1729056660' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:32:13.075772] [YB420AC531B8-00060B7DB46FAB68-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990814) Duplicate entry '586774769288417280-1729056660' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:32:13.075772] [YB420AC531B8-00060B7DB46FAB68-0-0] 2024-10-16 13:32:13.319 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056660, nodeCount=450, affectRows=450 2024-10-16 13:32:13.321 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056660, nodeCount=449, validCount=449 2024-10-16 13:32:13.452 INFO 4486 --- [ocp-schedules-22,3e9a1c5c1008e45c,3573aeaeff7ef166] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:32:13.994 INFO 4486 --- [prometheus-minute-schedule-1,fa18edbafa872346,0f21cb318f75c814] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:14.001 INFO 4486 --- [prometheus-minute-schedule-1,fa18edbafa872346,0f21cb318f75c814] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:14.358 INFO 4486 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056660, nodeCount=141, affectRows=141 2024-10-16 13:32:14.366 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056660, nodeCount=336, affectRows=336 2024-10-16 13:32:14.544 INFO 4486 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056660, nodeCount=141, validCount=141 2024-10-16 13:32:14.838 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056660, nodeCount=336, validCount=336 2024-10-16 13:32:15.034 INFO 4486 --- [ocp-async-2,b37ebb97c1f52f36,0ac750eaac7f14dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:15.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,583e5ba9c992e846,cfc021f233f58a81] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=583e5ba9c992e846, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:32:15.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,583e5ba9c992e846,cfc021f233f58a81] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=583e5ba9c992e846, duration=1 ms] 2024-10-16 13:32:15.717 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:16.994 INFO 4486 --- [prometheus-minute-schedule-1,5231681036fab5d0,29626a46d3cb37d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:17.001 INFO 4486 --- [prometheus-minute-schedule-1,5231681036fab5d0,29626a46d3cb37d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:17.002 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,93cdd70fec0c4bc6,2d57903978b1830d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=93cdd70fec0c4bc6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:17.003 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,93cdd70fec0c4bc6,2d57903978b1830d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=93cdd70fec0c4bc6, duration=4 ms] 2024-10-16 13:32:17.034 INFO 4486 --- [ocp-async-5,ea65fdb211ccb7a1,4a429d2f5e462ff9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:17.069 INFO 4486 --- [ocp-schedules-12,8681667f5126cf84,8a434879ad587a2f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:32:17.079 INFO 4486 --- [ocp-schedules-27,cb3edc8e35c878e1,8adcd672f9055013] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:32:17.180 INFO 4486 --- [ocp-schedules-15,98615afcde47b1b8,5e302841e2cc32bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:32:17.927 INFO 4486 --- [ocp-schedules-10,945782074d237944,253241be5716be5c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:32:17.975 WARN 4486 --- [scanner-13246554-1,945782074d237944,90ccc3066072e778] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:32:18.022 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9dd15a188c4cf0ea,c80d8e31c57d6aaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=9dd15a188c4cf0ea, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:32:18.025 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9dd15a188c4cf0ea,c80d8e31c57d6aaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=9dd15a188c4cf0ea, duration=10 ms] 2024-10-16 13:32:18.042 INFO 4486 --- [alarm-detect-11,945782074d237944,b63a5e590e043fca] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056705 ~ 1729056728 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056720, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056720, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056720=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:32:18.718 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:18.837 INFO 4486 --- [ocp-schedules-27,96e796ba935da921,fea54a5946447227] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:32:19.034 INFO 4486 --- [ocp-async-10,8a6db35e130a50aa,d0bc55862f0043a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:19.224 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d2d96f9cc8dd82de,9ef126bb1b41db4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d2d96f9cc8dd82de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:19.224 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d2d96f9cc8dd82de,9ef126bb1b41db4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d2d96f9cc8dd82de, duration=1 ms] 2024-10-16 13:32:19.994 INFO 4486 --- [prometheus-minute-schedule-1,2fdd072c6b19412e,12b734a8c59b92c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:20.001 INFO 4486 --- [prometheus-minute-schedule-1,2fdd072c6b19412e,12b734a8c59b92c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:20.253 INFO 4486 --- [ocp-schedules-3,efef0297011025c4,c87d7959731047cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:32:20.810 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:32:20.821 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,c2ec57d5fd125589,0642aa21a0c0ae43] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c2ec57d5fd125589, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:20.821 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,c2ec57d5fd125589,0642aa21a0c0ae43] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c2ec57d5fd125589, duration=2 ms] 2024-10-16 13:32:20.868 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 13:32:20.868 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 13:32:20.996 INFO 4486 --- [prometheus-cache-monitor1,d7b7a1d0203c3b80,ee6fd7fea65ef118] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:32:20.996 INFO 4486 --- [prometheus-cache-monitor1,d7b7a1d0203c3b80,ee6fd7fea65ef118] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:32:21.034 INFO 4486 --- [ocp-async-8,f29117a919139e78,fd79605bca14e669] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:21.194 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,052655af1798467a,3264a0a2acf12462] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=052655af1798467a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:21.194 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,052655af1798467a,3264a0a2acf12462] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=052655af1798467a, duration=1 ms] 2024-10-16 13:32:21.719 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:22.696 INFO 4486 --- [ocp-schedules-26,f41dc21e11de1643,97306d2637062687] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:32:22.985 INFO 4486 --- [ocp-async-2,e549e1390e992462,41f0dd580e6593b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:32:22.994 INFO 4486 --- [prometheus-minute-schedule-1,1c3e4544ada3fa2a,e4fef3a69d383c12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:23.001 INFO 4486 --- [prometheus-minute-schedule-1,1c3e4544ada3fa2a,e4fef3a69d383c12] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:23.035 INFO 4486 --- [ocp-async-4,8837c62ae7c9eb9a,214466e63aa76157] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:23.737 INFO 4486 --- [ocp-schedules-30,c25634d9e2ca00b6,c0754b71c8e67c8f] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:32:23.740 INFO 4486 --- [ocp-schedules-30,c25634d9e2ca00b6,c0754b71c8e67c8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:23.782 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:32:23.782 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.786 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:32:23.799 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.799 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:32:23.808 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.809 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:32:23.811 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:32:23.812 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.818 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.818 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:32:23.819 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:32:23.826 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.827 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:32:23.837 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.837 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.838 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:32:23.838 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:23.845 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:32:23.845 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.848 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.848 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.849 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:23.849 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:32:23.850 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:32:23.858 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.858 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:32:23.859 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.860 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:23.861 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.862 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:32:23.868 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.868 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:32:23.871 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.872 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:23.874 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.875 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:32:23.882 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.883 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:32:23.885 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.886 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:32:23.890 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.891 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:32:23.894 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.895 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.896 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.897 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.898 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.899 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.899 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.900 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:32:23.900 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.901 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.901 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:32:23.901 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.902 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.903 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.904 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.905 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.906 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.907 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.908 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.909 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.910 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.910 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.911 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:32:23.911 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.912 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.912 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.912 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.913 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:23.914 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.915 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.915 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.916 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.917 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:32:23.921 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.922 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:23.925 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:32:23.930 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.930 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.931 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.931 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:23.934 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.935 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.940 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.942 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.944 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.945 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:23.945 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.946 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.946 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:32:23.947 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.948 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.950 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.952 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.954 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.956 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:23.957 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:32:23.958 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.960 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.962 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:32:23.962 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:32:23.962 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.964 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.964 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.965 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:32:23.966 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.968 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:32:23.969 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.971 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.974 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.975 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.977 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.978 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.981 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.982 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.984 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.985 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.988 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.989 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:32:23.991 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.993 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:32:23.995 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:23.996 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:32:23.999 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:32:24.002 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:32:24.005 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:24.006 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:24.007 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:32:24.021 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:24.021 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:32:24.021 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:24.021 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:32:24.021 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:24.021 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:32:24.027 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:32:24.027 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:32:24.066 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:24.068 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:24.069 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:32:24.078 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:24.078 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:32:24.078 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:24.078 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:32:24.078 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:32:24.078 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:32:24.082 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:32:24.082 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:32:24.139 INFO 4486 --- [ocp-schedules-5,89aae6304c33ee95,277e2972d5bd8f4e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:32:24.379 INFO 4486 --- [ocp-schedules-7,dae6154f47033d1d,f376b6a11a44e032] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:32:24.379 INFO 4486 --- [ocp-schedules-7,dae6154f47033d1d,f376b6a11a44e032] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:32:24.411 INFO 4486 --- [Collect all tenants major compactions3,dae6154f47033d1d,eb5233f5e3ab99a7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:32:24.411 INFO 4486 --- [Collect all tenants major compactions1,dae6154f47033d1d,801d649d0da28669] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:32:24.411 INFO 4486 --- [Collect all tenants major compactions2,dae6154f47033d1d,d95ec7f4fed37e70] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:32:24.411 INFO 4486 --- [Collect all tenants major compactions4,dae6154f47033d1d,e474db854c7b667f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:32:24.411 INFO 4486 --- [Collect all tenants major compactions5,dae6154f47033d1d,89befdbf63cb130a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:32:24.429 INFO 4486 --- [Collect all tenants major compactions1,dae6154f47033d1d,801d649d0da28669] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:24.430 INFO 4486 --- [Collect all tenants major compactions2,dae6154f47033d1d,d95ec7f4fed37e70] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:24.430 INFO 4486 --- [Collect all tenants major compactions1,dae6154f47033d1d,801d649d0da28669] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:32:24.432 INFO 4486 --- [Collect all tenants major compactions2,dae6154f47033d1d,d95ec7f4fed37e70] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:32:24.433 INFO 4486 --- [Collect all tenants major compactions3,dae6154f47033d1d,eb5233f5e3ab99a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:24.434 INFO 4486 --- [Collect all tenants major compactions1,dae6154f47033d1d,801d649d0da28669] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:32:24.435 INFO 4486 --- [Collect all tenants major compactions2,dae6154f47033d1d,d95ec7f4fed37e70] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:32:24.435 INFO 4486 --- [Collect all tenants major compactions3,dae6154f47033d1d,eb5233f5e3ab99a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:32:24.439 INFO 4486 --- [Collect all tenants major compactions1,dae6154f47033d1d,801d649d0da28669] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:32:24.440 INFO 4486 --- [Collect all tenants major compactions2,dae6154f47033d1d,d95ec7f4fed37e70] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:32:24.441 INFO 4486 --- [Collect all tenants major compactions5,dae6154f47033d1d,89befdbf63cb130a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:24.441 INFO 4486 --- [Collect all tenants major compactions4,dae6154f47033d1d,e474db854c7b667f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:24.442 INFO 4486 --- [Collect all tenants major compactions3,dae6154f47033d1d,eb5233f5e3ab99a7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:32:24.442 INFO 4486 --- [Collect all tenants major compactions3,dae6154f47033d1d,eb5233f5e3ab99a7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:32:24.443 INFO 4486 --- [Collect all tenants major compactions5,dae6154f47033d1d,89befdbf63cb130a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:32:24.443 INFO 4486 --- [Collect all tenants major compactions4,dae6154f47033d1d,e474db854c7b667f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:32:24.446 INFO 4486 --- [Collect all tenants major compactions4,dae6154f47033d1d,e474db854c7b667f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:32:24.446 INFO 4486 --- [Collect all tenants major compactions4,dae6154f47033d1d,e474db854c7b667f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:32:24.448 INFO 4486 --- [Collect all tenants major compactions5,dae6154f47033d1d,89befdbf63cb130a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:32:24.448 INFO 4486 --- [Collect all tenants major compactions5,dae6154f47033d1d,89befdbf63cb130a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:32:24.449 INFO 4486 --- [ocp-schedules-7,dae6154f47033d1d,f376b6a11a44e032] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:32:24.720 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:25.035 INFO 4486 --- [ocp-async-1,3affc7eaa6a0d23f,292fc09165626c29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:25.994 INFO 4486 --- [prometheus-minute-schedule-1,8846669fce5cb17a,f77315ca99d37a6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:26.001 INFO 4486 --- [prometheus-minute-schedule-1,8846669fce5cb17a,f77315ca99d37a6f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:26.822 INFO 4486 --- [ocp-schedules-29,58806b17b296a5ca,bab4fdee6d47897b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:32:26.822+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:32:26.824 INFO 4486 --- [ocp-schedules-29,58806b17b296a5ca,bab4fdee6d47897b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:32:26.825 INFO 4486 --- [ocp-schedules-29,58806b17b296a5ca,bab4fdee6d47897b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:32:26.825 INFO 4486 --- [ocp-schedules-29,58806b17b296a5ca,bab4fdee6d47897b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:32:27.009 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,650b87bc54c33185,a5d7a91f4c6f1baa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=650b87bc54c33185, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:27.009 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,650b87bc54c33185,a5d7a91f4c6f1baa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=650b87bc54c33185, duration=1 ms] 2024-10-16 13:32:27.035 INFO 4486 --- [ocp-async-8,9e07c05408dc07b5,8b1a8d65cafa07da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:27.721 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:28.106 INFO 4486 --- [ocp-schedules-23,2393b06648a303aa,d5eac92141bc8fda] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:32:28.152 WARN 4486 --- [scanner-13246798-1,2393b06648a303aa,4c6fb21c634e967b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:32:28.177 INFO 4486 --- [alarm-detect-14,2393b06648a303aa,223eea9cb33f7512] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:28.182 INFO 4486 --- [alarm-detect-14,2393b06648a303aa,223eea9cb33f7512] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:28.186 INFO 4486 --- [alarm-detect-14,2393b06648a303aa,223eea9cb33f7512] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:28.188 INFO 4486 --- [alarm-detect-2,2393b06648a303aa,9b02551f9324925f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:28.198 INFO 4486 --- [alarm-detect-2,2393b06648a303aa,9b02551f9324925f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:28.198 INFO 4486 --- [alarm-detect-15,2393b06648a303aa,1dafee7a163ecb8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:28.203 INFO 4486 --- [alarm-detect-2,2393b06648a303aa,9b02551f9324925f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:28.204 INFO 4486 --- [alarm-detect-15,2393b06648a303aa,1dafee7a163ecb8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:28.214 INFO 4486 --- [alarm-detect-15,2393b06648a303aa,1dafee7a163ecb8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:28.259 INFO 4486 --- [alarm-detect-18,2393b06648a303aa,4f0cfe460ef5c689] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056720 ~ 1729056738 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056735, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056735, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056735=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:32:28.300 INFO 4486 --- [alarm-detect-15,2393b06648a303aa,af12c414c6a916a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:28.305 INFO 4486 --- [alarm-detect-15,2393b06648a303aa,af12c414c6a916a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:28.309 INFO 4486 --- [alarm-detect-15,2393b06648a303aa,af12c414c6a916a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:28.994 INFO 4486 --- [prometheus-minute-schedule-1,8df7d70d5648d373,a8bd3b3db4fad3b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:28.999 INFO 4486 --- [prometheus-minute-schedule-1,8df7d70d5648d373,a8bd3b3db4fad3b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:29.035 INFO 4486 --- [ocp-async-2,26f026fc5094c8d8,88283af57a2c20f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:29.065 INFO 4486 --- [ocp-schedules-16,d9dd51771f920a57,bb07a3ad2db33f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:32:29.065 INFO 4486 --- [ocp-schedules-16,b3f44f2ec63a2995,501af34be360a5f4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:32:29.067 INFO 4486 --- [ocp-schedules-16,b3f44f2ec63a2995,501af34be360a5f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:29.099 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.100 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:29.105 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.106 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:29.132 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.133 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:29.135 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.140 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.140 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:32:29.170 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.171 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.172 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:29.173 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:32:29.176 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.179 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:32:29.181 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:32:29.195 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.195 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:32:29.210 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.218 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.219 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.220 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:32:29.220 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:32:29.225 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:32:29.228 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dffa5774c4c404de,8bd5858d53fe000d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dffa5774c4c404de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:29.229 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dffa5774c4c404de,8bd5858d53fe000d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dffa5774c4c404de, duration=2 ms] 2024-10-16 13:32:29.229 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:29.231 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:29.234 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.236 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.237 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:32:29.258 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:32:29.272 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:32:29.272 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:32:29.286 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.287 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.288 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:29.292 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.295 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:32:29.299 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:32:29.325 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.325 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:32:29.341 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:32:29.351 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.356 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.356 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:32:29.356 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.356 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:32:29.367 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:32:29.367 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:32:29.385 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.385 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:32:29.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:32:29.442 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.442 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:32:29.470 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:29.498 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.500 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.501 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:32:29.506 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:32:29.510 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:32:29.513 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:32:29.515 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:32:29.539 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:32:29.673 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:32:29.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:29.706 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:29.707 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:32:29.736 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:32:29.736 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:32:30.102 INFO 4486 --- [prometheus-second-schedule-1,7f0444ae85873433,0521d8314d1dddbf] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:32:30.258 INFO 4486 --- [ocp-schedules-17,3f32b46435622fe6,d82ffc5913190875] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:32:30.722 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:30.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d348f66b943e963a,2a475bd0ddc9e332] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d348f66b943e963a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:30.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d348f66b943e963a,2a475bd0ddc9e332] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d348f66b943e963a, duration=2 ms] 2024-10-16 13:32:31.035 INFO 4486 --- [ocp-async-4,5c2e00351e861751,b0aff29dc6f0eb07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:31.198 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,d47220cc315bc6f7,9b35bfebf3ecce6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d47220cc315bc6f7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:31.198 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,d47220cc315bc6f7,9b35bfebf3ecce6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d47220cc315bc6f7, duration=1 ms] 2024-10-16 13:32:31.994 INFO 4486 --- [prometheus-minute-schedule-1,1e9ce52f925a9059,49889bcf50cc2414] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:32.000 INFO 4486 --- [prometheus-minute-schedule-1,1e9ce52f925a9059,49889bcf50cc2414] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:32.985 INFO 4486 --- [ocp-async-1,fa05a2f6d599d3b6,acd24ba732b88bfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:32:33.036 INFO 4486 --- [ocp-async-9,2f3e776fe909c390,fae50655e13bb4a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:33.723 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:34.994 INFO 4486 --- [prometheus-minute-schedule-1,6a7fa12cd81a42ab,d82bb2efc78fa0a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:35.002 INFO 4486 --- [prometheus-minute-schedule-1,6a7fa12cd81a42ab,d82bb2efc78fa0a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:35.036 INFO 4486 --- [ocp-async-2,f62704b98e741b35,555b6dac79323b53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:36.724 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:37.017 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,9d93368b9d7b0365,78d2db1f4087d97e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9d93368b9d7b0365, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:37.018 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,9d93368b9d7b0365,78d2db1f4087d97e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9d93368b9d7b0365, duration=3 ms] 2024-10-16 13:32:37.036 INFO 4486 --- [ocp-async-4,921b2f34d6e9a30f,2759d9fc3dcffcfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:37.063 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:32:37.064 INFO 4486 --- [ocp-schedules-15,6362d1c1dfbee82f,2684a486950f5437] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:32:37.066 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:32:37.066 INFO 4486 --- [ocp-schedules-28,9c56fdf1c4236b3f,70e4407f5652990f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:32:37.068 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:32:37.070 INFO 4486 --- [ocp-schedules-28,9c56fdf1c4236b3f,70e4407f5652990f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:37.070 INFO 4486 --- [ocp-schedules-15,6362d1c1dfbee82f,2684a486950f5437] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:32:37.081 INFO 4486 --- [ocp-schedules-7,5fc530a8dccb7e71,8034d88623e8711f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:32:37.083 INFO 4486 --- [ocp-schedules-7,5fc530a8dccb7e71,8034d88623e8711f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:32:37.090 INFO 4486 --- [ocp-schedules-9,39bd6c156bcd4613,0610f39b2dbccee4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:32:37.090 INFO 4486 --- [ocp-schedules-20,f0c2c5c00fa3f299,de370f447531fddd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:32:37.091 INFO 4486 --- [host-check-clockdiff1,39bd6c156bcd4613,fa5093d560718067] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:32:37.091 INFO 4486 --- [host-check-clockdiff2,39bd6c156bcd4613,69e01f32f61b8bd5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:32:37.091 INFO 4486 --- [host-check-clockdiff3,39bd6c156bcd4613,5839f433e29bd918] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:32:37.091 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:32:37.092 INFO 4486 --- [ocp-schedules-1,72f34d268a074fc2,eaab0a2c2a7b9be3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:32:37.092 INFO 4486 --- [host-check-clockdiff4,39bd6c156bcd4613,82b0dcd4e49e696d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:32:37.093 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:32:37.093 INFO 4486 --- [host-check-agent-status1,72f34d268a074fc2,5692568c25500cb0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:32:37.094 INFO 4486 --- [host-check-agent-status2,72f34d268a074fc2,ee6c9e83b891535f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:32:37.094 INFO 4486 --- [host-check-agent-status3,72f34d268a074fc2,a5b02b5cd5c196a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:32:37.094 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:32:37.095 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:32:37.095 INFO 4486 --- [host-check-agent-status4,72f34d268a074fc2,9ccc28923d559331] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:32:37.119 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:37.119 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:37.120 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:37.126 INFO 4486 --- [host-check-agent-status4,72f34d268a074fc2,9ccc28923d559331] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:37.126 INFO 4486 --- [host-check-agent-status3,72f34d268a074fc2,a5b02b5cd5c196a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:37.126 INFO 4486 --- [host-check-clockdiff2,39bd6c156bcd4613,69e01f32f61b8bd5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056757 3 3 , error= 2024-10-16 13:32:37.126 INFO 4486 --- [host-check-clockdiff4,39bd6c156bcd4613,82b0dcd4e49e696d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056757 3 3 , error= 2024-10-16 13:32:37.126 INFO 4486 --- [host-check-clockdiff3,39bd6c156bcd4613,5839f433e29bd918] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056757 3 3 , error= 2024-10-16 13:32:37.128 INFO 4486 --- [host-check-agent-status1,72f34d268a074fc2,5692568c25500cb0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:37.129 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:37.129 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:37.129 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:37.129 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:37.129 INFO 4486 --- [host-check-clockdiff1,39bd6c156bcd4613,fa5093d560718067] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056757 0 0 , error= 2024-10-16 13:32:37.132 INFO 4486 --- [host-check-agent-status3,72f34d268a074fc2,a5b02b5cd5c196a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:37.132 INFO 4486 --- [host-check-agent-status4,72f34d268a074fc2,9ccc28923d559331] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:37.133 INFO 4486 --- [host-check-agent-status2,72f34d268a074fc2,ee6c9e83b891535f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:37.133 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:37.133 INFO 4486 --- [host-check-agent-status1,72f34d268a074fc2,5692568c25500cb0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:37.134 INFO 4486 --- [host-check-agent-status2,72f34d268a074fc2,ee6c9e83b891535f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:37.137 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.137 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:32:37.138 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:32:37.159 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.160 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:32:37.161 INFO 4486 --- [host-check-agent-status3,72f34d268a074fc2,a5b02b5cd5c196a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:32:37.161 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:32:37.161 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:32:37.162 INFO 4486 --- [host-check-agent-status4,72f34d268a074fc2,9ccc28923d559331] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:32:37.162 INFO 4486 --- [host-check-agent-status1,72f34d268a074fc2,5692568c25500cb0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:32:37.162 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:32:37.163 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:32:37.164 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:32:37.164 INFO 4486 --- [host-check-agent-status2,72f34d268a074fc2,ee6c9e83b891535f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:32:37.165 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:32:37.164712322+08:00 2024-10-16 13:32:37.165 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:32:37.168283499+08:00 2024-10-16 13:32:37.165 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:32:37.168065114+08:00 2024-10-16 13:32:37.166 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:32:37.168562775+08:00 2024-10-16 13:32:37.166 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:32:37.165+08:00 2024-10-16 13:32:37.166 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:32:37.165+08:00 2024-10-16 13:32:37.166 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:32:37.165+08:00 2024-10-16 13:32:37.167 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.167 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:32:37.166+08:00 2024-10-16 13:32:37.168 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:32:37.168 INFO 4486 --- [host-check-agent-status3,72f34d268a074fc2,a5b02b5cd5c196a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:32:37.168 INFO 4486 --- [host-check-agent-status3,72f34d268a074fc2,a5b02b5cd5c196a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:32:37.168 INFO 4486 --- [host-check-agent-status4,72f34d268a074fc2,9ccc28923d559331] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:32:37.168 INFO 4486 --- [host-check-agent-status4,72f34d268a074fc2,9ccc28923d559331] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:32:37.168 INFO 4486 --- [host-check-agent-status3,72f34d268a074fc2,a5b02b5cd5c196a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:32:37.168 INFO 4486 --- [host-check-agent-status4,72f34d268a074fc2,9ccc28923d559331] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:32:37.169 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.169 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:32:37.165+08:00 and status AVAILABLE 2024-10-16 13:32:37.169 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:32:37.165+08:00 and status AVAILABLE 2024-10-16 13:32:37.169 INFO 4486 --- [host-check-agent-status1,72f34d268a074fc2,5692568c25500cb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:32:37.169 INFO 4486 --- [host-check-agent-status1,72f34d268a074fc2,5692568c25500cb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:32:37.169 INFO 4486 --- [host-check-agent-status1,72f34d268a074fc2,5692568c25500cb0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:32:37.169 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.169 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:32:37.165+08:00 and status AVAILABLE 2024-10-16 13:32:37.170 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:32:37.166+08:00 and status AVAILABLE 2024-10-16 13:32:37.169 INFO 4486 --- [host-check-agent-status2,72f34d268a074fc2,ee6c9e83b891535f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:32:37.170 INFO 4486 --- [host-check-agent-status2,72f34d268a074fc2,ee6c9e83b891535f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:32:37.170 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:32:37.170 INFO 4486 --- [host-check-agent-status2,72f34d268a074fc2,ee6c9e83b891535f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:32:37.171 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:32:37.172 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:32:37.172 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:32:37.172 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:32:37.175 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.179 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:32:37.181 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:37.181 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:37.181 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:37.182 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:37.182 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:37.182 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:37.183 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:37.183 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:37.185 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.185 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.185 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.185 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.185 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.185 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.187 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.188 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:37.202 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.203 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:37.206 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:32:37.208 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:37.213 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:37.213 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:37.215 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:32:37.218 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.219 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:32:37.221 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.222 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:32:37.231 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.232 INFO 4486 --- [ocp-schedules-5,51be9dde534bfb2f,173f5b1c0805193d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:32:37.243 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:32:37.249 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:32:37.251 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:32:37.255 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:32:37.261 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:32:37.263 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:32:37.272 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:32:37.275 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:37.279 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:37.280 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:32:37.282 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:37.295 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:32:37.305 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:32:37.310 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:32:37.316 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:32:37.321 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:32:37.333 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:37.337 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:37.338 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:32:37.350 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:32:37.350 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:32:37.361 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:32:37.362 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:32:37.368 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:37.370 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:32:37.372 INFO 4486 --- [host-check-host-status3,f0c2c5c00fa3f299,e4f96d20aeaee333] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:32:37.378 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:32:37.381 INFO 4486 --- [host-check-host-status4,f0c2c5c00fa3f299,61227924f1a9aa3e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:32:37.409 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:32:37.419 INFO 4486 --- [host-check-host-status2,f0c2c5c00fa3f299,7c21a97dfea8b667] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:32:37.421 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:37.423 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.424 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:32:37.425 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.426 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.427 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:32:37.438 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:32:37.438 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.438 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.438 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.438 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.438 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.439 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.448 INFO 4486 --- [host-check-host-status1,f0c2c5c00fa3f299,c8c797551f6383a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:32:37.466 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:32:37.522 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:32:37.570 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:37.585 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:37.589 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.590 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:32:37.591 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:32:37.597 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.598 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:32:37.599 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.600 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.601 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:32:37.611 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.611 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.611 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.611 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.611 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.612 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.619 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:32:37.838 INFO 4486 --- [ocp-schedules-8,156d1d516feba951,1f59288e902f6a53] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-10-16 13:32:37.838 INFO 4486 --- [ocp-schedules-8,156d1d516feba951,1f59288e902f6a53] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-10-16 13:32:37.838 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:32:37.884 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:37.886 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.889 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:32:37.891 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.891 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.896 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:32:37.911 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.911 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.911 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.911 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.911 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:37.912 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:37.929 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:32:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:32:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:32:37.994 INFO 4486 --- [prometheus-minute-schedule-1,b458aafa8bf1741e,495ed6397a8fbee4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:38.001 INFO 4486 --- [prometheus-minute-schedule-1,b458aafa8bf1741e,495ed6397a8fbee4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:38.145 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:32:38.197 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:38.199 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:38.200 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:32:38.201 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:38.201 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:38.202 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:32:38.213 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:38.213 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:38.213 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:38.213 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:38.213 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:32:38.213 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:38.279 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:32:38.338 INFO 4486 --- [ocp-schedules-22,0c453f8d034a2732,fc6a3f6761026230] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:32:38.374 WARN 4486 --- [scanner-13247135-1,0c453f8d034a2732,864d66c97d983878] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:32:38.433 INFO 4486 --- [ocp-schedules-5,1f35b1d47f621e2e,e6be31c8560f1338] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:32:38.434 INFO 4486 --- [ocp-schedules-5,1f35b1d47f621e2e,e6be31c8560f1338] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:32:38.435 INFO 4486 --- [ocp-schedules-5,1f35b1d47f621e2e,e6be31c8560f1338] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:32:38.550 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:32:38.282156, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:32:38.551 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:32:38.282156, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:32:38.593 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:32:38.282156} } 2024-10-16 13:32:38.669 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:32:38.674 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:32:38.677 INFO 4486 --- [ocp-schedules-13,0240aab4139b3682,8fc1b18a705cc3a9] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:32:38.842 INFO 4486 --- [ocp-schedules-17,a23db7709fbdab86,836ae85bada301ff] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:32:39.036 INFO 4486 --- [ocp-async-9,0a3c0ceebcd368fa,f9325305dc1b631b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:39.246 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2bc87ac1ee766bb6,b1f0d423140d8069] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2bc87ac1ee766bb6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:39.246 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2bc87ac1ee766bb6,b1f0d423140d8069] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2bc87ac1ee766bb6, duration=1 ms] 2024-10-16 13:32:39.725 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:40.006 INFO 4486 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:32:40.007 INFO 4486 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:32:40.028 INFO 4486 --- [Check obproxy2,50d0da6913175bbf,e86b2bd97b2b3627] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:32:40.028 INFO 4486 --- [Check obproxy1,b0af2a568b2899cf,b78cec03dd9dab41] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:32:40.028 INFO 4486 --- [Check obproxy1,b0af2a568b2899cf,b78cec03dd9dab41] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:40.028 INFO 4486 --- [Check obproxy2,50d0da6913175bbf,e86b2bd97b2b3627] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:40.031 INFO 4486 --- [Collect obproxy1,2ba7134a1e262fd9,4570f438e9d35f26] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:32:40.031 INFO 4486 --- [Collect obproxy2,9797cc90f97349aa,0d64a74550ad4882] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:32:40.031 INFO 4486 --- [Collect obproxy1,2ba7134a1e262fd9,4570f438e9d35f26] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:40.031 INFO 4486 --- [Collect obproxy2,9797cc90f97349aa,0d64a74550ad4882] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:32:40.052 INFO 4486 --- [Check obproxy2,50d0da6913175bbf,e86b2bd97b2b3627] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:32:40.052 INFO 4486 --- [Check obproxy2,50d0da6913175bbf,e86b2bd97b2b3627] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:32:40.053 INFO 4486 --- [Collect obproxy1,2ba7134a1e262fd9,4570f438e9d35f26] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:32:40.055 INFO 4486 --- [Collect obproxy2,9797cc90f97349aa,0d64a74550ad4882] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:32:40.057 INFO 4486 --- [Check obproxy1,b0af2a568b2899cf,b78cec03dd9dab41] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:32:40.057 INFO 4486 --- [Check obproxy1,b0af2a568b2899cf,b78cec03dd9dab41] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:32:40.059 INFO 4486 --- [Collect obproxy1,2ba7134a1e262fd9,4570f438e9d35f26] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:32:40.061 INFO 4486 --- [Collect obproxy2,9797cc90f97349aa,0d64a74550ad4882] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:32:40.062 INFO 4486 --- [Collect obproxy1,2ba7134a1e262fd9,4570f438e9d35f26] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:32:40.066 INFO 4486 --- [Collect obproxy2,9797cc90f97349aa,0d64a74550ad4882] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:32:40.264 INFO 4486 --- [ocp-schedules-8,7b3f00cce947a2b9,c4958fdad4cc4348] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:32:40.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5d18721fd12b98be,5aeaec433b1e420b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5d18721fd12b98be, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:40.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5d18721fd12b98be,5aeaec433b1e420b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5d18721fd12b98be, duration=1 ms] 2024-10-16 13:32:40.994 INFO 4486 --- [prometheus-minute-schedule-1,f629d89c126cefd6,16c01521b1e6d231] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:40.995 INFO 4486 --- [prometheus-hour-schedule-1,804e11d485bfc3c7,7752d9eba93d4b2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:32:40.999 INFO 4486 --- [prometheus-minute-schedule-1,f629d89c126cefd6,16c01521b1e6d231] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:41.037 INFO 4486 --- [ocp-async-2,8d80bb082bc0f111,a63362f276e6315c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:41.203 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5c99533d58c4c124,b979acf9cf24641c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5c99533d58c4c124, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:41.203 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5c99533d58c4c124,b979acf9cf24641c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5c99533d58c4c124, duration=1 ms] 2024-10-16 13:32:42.080 INFO 4486 --- [ocp-schedules-3,01a2708a8e51d936,656a3179540db5a1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:32:42.726 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:42.986 INFO 4486 --- [ocp-async-3,27678f72e5b96c42,8e2e2fd13fac378a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:32:43.037 INFO 4486 --- [ocp-async-4,b2a8e3bb0f2e5e1e,95d2139f99dcb29e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:43.994 INFO 4486 --- [prometheus-minute-schedule-1,40275b928373e481,6bc8b0143dbb3587] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:44.003 INFO 4486 --- [prometheus-minute-schedule-1,40275b928373e481,6bc8b0143dbb3587] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:45.037 INFO 4486 --- [ocp-async-8,a520a7b873ea8b8d,59dc41990bcbc896] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:45.437 INFO 4486 --- [ocp-schedules-12,08125430f3b1719f,1b701497a41ee983] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:32:45.442 INFO 4486 --- [ocp-schedules-12,08125430f3b1719f,1b701497a41ee983] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:32:45.442 INFO 4486 --- [ocp-schedules-12,08125430f3b1719f,1b701497a41ee983] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:32:45.453 INFO 4486 --- [ocp-schedules-5,84f629e2ec47102e,2b9f6f08fe3d77d9] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:32:45.453 INFO 4486 --- [ocp-schedules-5,84f629e2ec47102e,2b9f6f08fe3d77d9] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:32:45.727 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:46.411 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1da0c748d03c5b02,d7abdaf7f258bfb1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=1da0c748d03c5b02, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:32:46.412 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1da0c748d03c5b02,d7abdaf7f258bfb1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=1da0c748d03c5b02, duration=2 ms] 2024-10-16 13:32:46.994 INFO 4486 --- [prometheus-minute-schedule-1,58906f840d0ab776,d0ec3b6d67d39cf7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:47.000 INFO 4486 --- [prometheus-minute-schedule-1,58906f840d0ab776,d0ec3b6d67d39cf7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:47.026 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6e59982cfb8f8bdb,a238c4436aa9c1b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6e59982cfb8f8bdb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:47.026 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6e59982cfb8f8bdb,a238c4436aa9c1b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6e59982cfb8f8bdb, duration=1 ms] 2024-10-16 13:32:47.037 INFO 4486 --- [ocp-async-7,84f11801b01dbcac,f10eed88bf58f016] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:47.072 INFO 4486 --- [ocp-schedules-28,9c56fdf1c4236b3f,70e4407f5652990f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:32:47.083 INFO 4486 --- [ocp-schedules-9,b22c6ec4765583a4,c886fb8585530e35] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:32:47.164 INFO 4486 --- [ocp-schedules-7,5fc530a8dccb7e71,8034d88623e8711f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:32:48.502 INFO 4486 --- [ocp-schedules-26,dd7083855b817585,64d6186bef6e61e7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:32:48.527 WARN 4486 --- [scanner-13247368-1,dd7083855b817585,5458456237a7c409] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:32:48.565 INFO 4486 --- [alarm-detect-10,dd7083855b817585,52a068f22ac0dafa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:48.571 INFO 4486 --- [alarm-detect-10,dd7083855b817585,52a068f22ac0dafa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:48.577 INFO 4486 --- [alarm-detect-10,dd7083855b817585,52a068f22ac0dafa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:32:48.589 INFO 4486 --- [alarm-detect-9,dd7083855b817585,bf3e65b9c5bec968] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:48.589 INFO 4486 --- [alarm-detect-7,dd7083855b817585,50a3ef0a1cce4b4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:48.594 INFO 4486 --- [alarm-detect-9,dd7083855b817585,bf3e65b9c5bec968] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:48.595 INFO 4486 --- [alarm-detect-7,dd7083855b817585,50a3ef0a1cce4b4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:48.598 INFO 4486 --- [alarm-detect-9,dd7083855b817585,bf3e65b9c5bec968] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:32:48.601 INFO 4486 --- [alarm-detect-7,dd7083855b817585,50a3ef0a1cce4b4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:32:48.632 INFO 4486 --- [alarm-detect-14,dd7083855b817585,a3a0f876f6ae6110] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056735 ~ 1729056758 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056750, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056750, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056750=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:32:48.677 INFO 4486 --- [alarm-detect-19,dd7083855b817585,2c62a7b65c382bd1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:48.683 INFO 4486 --- [alarm-detect-19,dd7083855b817585,2c62a7b65c382bd1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:48.689 INFO 4486 --- [alarm-detect-19,dd7083855b817585,2c62a7b65c382bd1] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:32:48.727 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:49.037 INFO 4486 --- [ocp-async-1,64c0dfe0084a3d7b,e0c16ba2c5073f49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:49.250 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c5d47ee4ed764599,689f9f6a74c2405f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c5d47ee4ed764599, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:49.251 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c5d47ee4ed764599,689f9f6a74c2405f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c5d47ee4ed764599, duration=2 ms] 2024-10-16 13:32:49.845 INFO 4486 --- [ocp-schedules-18,c789846b5759df7d,ab085463e45dd44c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:32:49.845 INFO 4486 --- [ocp-schedules-18,c789846b5759df7d,ab085463e45dd44c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:32:49.845 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:32:49.845 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:32:49.845 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:32:49.845 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:32:49.845 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:32:49.851 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:49.851 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:32:49.851 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:49.851 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:49.851 INFO 4486 --- [ocp-schedules-18,c789846b5759df7d,ab085463e45dd44c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:32:49.852 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:49.853 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:32:49.853 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:49.853 INFO 4486 --- [ocp-schedules-18,c789846b5759df7d,ab085463e45dd44c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:32:49.853 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:49.855 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:49.856 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:49.877 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:32:49.879 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:32:49.881 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:32:49.882 INFO 4486 --- [ocp-schedules-18,c789846b5759df7d,ab085463e45dd44c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:32:49.883 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:32:49.883 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:32:49.886 INFO 4486 --- [ocp-schedules-18,c789846b5759df7d,ab085463e45dd44c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:32:49.994 INFO 4486 --- [prometheus-minute-schedule-1,71b600291aaca03a,2d72c3d682dfc9b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:50.001 INFO 4486 --- [prometheus-minute-schedule-1,71b600291aaca03a,2d72c3d682dfc9b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:50.269 INFO 4486 --- [ocp-schedules-3,89310848dd036bcc,9a823559877e912c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:32:50.359 INFO 4486 --- [ocp-schedules-17,5a91f1daabb3f1b2,e840832613be1f9d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:32:50.359 INFO 4486 --- [ocp-schedules-17,5a91f1daabb3f1b2,e840832613be1f9d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:32:50.365 INFO 4486 --- [ocp-schedules-17,5a91f1daabb3f1b2,e840832613be1f9d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:32:50.365 INFO 4486 --- [ocp-schedules-17,5a91f1daabb3f1b2,e840832613be1f9d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:32:50.369 INFO 4486 --- [ocp-schedules-17,5a91f1daabb3f1b2,e840832613be1f9d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:32:50.369 INFO 4486 --- [ocp-schedules-17,5a91f1daabb3f1b2,e840832613be1f9d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:32:50.837 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f91412477177625f,4b89ee6f56321ba6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f91412477177625f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:50.837 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f91412477177625f,4b89ee6f56321ba6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f91412477177625f, duration=1 ms] 2024-10-16 13:32:51.038 INFO 4486 --- [ocp-async-6,bec88d8d073a3135,c631ee2aa8d02b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:51.207 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,18d67d393b006f6c,190c2f27db0fb94b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=18d67d393b006f6c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:51.207 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,18d67d393b006f6c,190c2f27db0fb94b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=18d67d393b006f6c, duration=1 ms] 2024-10-16 13:32:51.728 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:52.986 INFO 4486 --- [ocp-async-5,a46fe843363e4436,cf36caac6bd05598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:32:52.994 INFO 4486 --- [prometheus-minute-schedule-1,184cd1b3802354e6,ee9d08919a231376] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:53.002 INFO 4486 --- [prometheus-minute-schedule-1,184cd1b3802354e6,ee9d08919a231376] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:53.038 INFO 4486 --- [ocp-async-7,450473775289cc5a,aa22c3c6afb91a18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:54.729 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:55.038 INFO 4486 --- [ocp-async-10,5847c268265ba51f,3bafa731ca5d7a19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:55.994 INFO 4486 --- [prometheus-minute-schedule-1,bd8b5faa1eebb819,83d035e11c6f90e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:56.000 INFO 4486 --- [prometheus-minute-schedule-1,bd8b5faa1eebb819,83d035e11c6f90e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:57.031 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,eda4d4352dd31d52,470480047b001f6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eda4d4352dd31d52, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:57.031 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,eda4d4352dd31d52,470480047b001f6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eda4d4352dd31d52, duration=2 ms] 2024-10-16 13:32:57.038 INFO 4486 --- [ocp-async-9,00f9a40e23974563,daf34c2102fec583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:57.729 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:32:58.719 INFO 4486 --- [ocp-schedules-21,11bad92d00530486,42e52b7db2e467f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:32:58.745 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:32:58.774 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:58.778 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:58.779 WARN 4486 --- [scanner-13247729-1,11bad92d00530486,ffd74737b56b576c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:32:58.820 INFO 4486 --- [alarm-detect-9,11bad92d00530486,77de181fe7fb4d44] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056750 ~ 1729056768 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056765, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056765, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056765=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:32:58.828 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:32:58.832 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:32:58.847 INFO 4486 --- [ocp-schedules-6,9b2137580175bd04,76747c183b1e76a0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:32:58.878 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:32:58.884 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:32:58.917 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:32:58.919 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:32:58.945 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:32:58.947 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:32:58.974 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:32:58.975 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:32:58.994 INFO 4486 --- [prometheus-minute-schedule-1,5dfe83188b5b630e,cb3aea6a942cffea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056780, interval=60 2024-10-16 13:32:59.001 INFO 4486 --- [prometheus-minute-schedule-1,5dfe83188b5b630e,cb3aea6a942cffea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:32:59.002 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:32:59.004 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:32:59.038 INFO 4486 --- [ocp-async-5,3abdf745c082926b,c9676397f301ac15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:32:59.045 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:32:59.047 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:32:59.072 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:32:59.074 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:32:59.102 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:32:59.104 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:32:59.127 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:32:59.128 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:32:59.153 INFO 4486 --- [ocp-schedules-12,83ff9c925956fdad,996e3cf56d51d021] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:32:59.256 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,880dcdd513ca5f7c,f0d0589a1e39b324] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=880dcdd513ca5f7c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:32:59.256 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,880dcdd513ca5f7c,f0d0589a1e39b324] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=880dcdd513ca5f7c, duration=1 ms] 2024-10-16 13:32:59.736 INFO 4486 --- [ocp-schedules-16,b3f44f2ec63a2995,501af34be360a5f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:32:59.737 INFO 4486 --- [ocp-schedules-16,bc12cc82494ea965,c56479a7a5b4325b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:32:59.738 INFO 4486 --- [ocp-schedules-16,bc12cc82494ea965,c56479a7a5b4325b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:32:59.769 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.770 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:59.778 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.779 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:59.808 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:59.809 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.809 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.809 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:32:59.810 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:59.841 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:59.842 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.842 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:59.843 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:32:59.843 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.846 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:32:59.847 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:32:59.858 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:59.858 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:32:59.869 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:32:59.878 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:59.878 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.879 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:32:59.880 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:32:59.885 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:32:59.891 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:32:59.893 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:32:59.921 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:32:59.923 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:59.924 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.925 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:32:59.946 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:32:59.946 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:32:59.973 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:32:59.975 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.976 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:32:59.979 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:32:59.982 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:32:59.986 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:32:59.998 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:33:00.008 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:00.009 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:00.010 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:33:00.013 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:00.013 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:33:00.019 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:33:00.019 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:33:00.037 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:00.037 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:33:00.062 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:00.062 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:33:00.089 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:00.090 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:33:00.116 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:00.116 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:33:00.144 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:00.174 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:00.175 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:00.176 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:33:00.181 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:33:00.185 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:33:00.190 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:33:00.192 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:33:00.215 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:33:00.275 INFO 4486 --- [ocp-schedules-9,7a83e0abf62627db,3a20ae1615711863] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:33:00.358 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:33:00.385 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:00.387 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:00.388 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:33:00.420 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:33:00.420 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:33:00.730 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:00.843 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a21b5d1b7c78785e,c16355d56c406599] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a21b5d1b7c78785e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:00.843 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a21b5d1b7c78785e,c16355d56c406599] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a21b5d1b7c78785e, duration=3 ms] 2024-10-16 13:33:01.039 INFO 4486 --- [ocp-async-1,3322b05d040de6df,545607eb0d11c144] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:01.050 INFO 4486 --- [ocp-schedules-14,c380369061409755,366fbb4c7b9617d0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:33:01.050+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:33:01.053 INFO 4486 --- [ocp-schedules-14,c380369061409755,366fbb4c7b9617d0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:33:01.064 INFO 4486 --- [ocp-schedules-14,c380369061409755,366fbb4c7b9617d0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:33:01.076 INFO 4486 --- [ocp-schedules-14,c380369061409755,366fbb4c7b9617d0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:33:01.211 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c5ce1c64919826a5,224ae22d68e7c466] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c5ce1c64919826a5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:01.211 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c5ce1c64919826a5,224ae22d68e7c466] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c5ce1c64919826a5, duration=1 ms] 2024-10-16 13:33:01.795 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6ad963b1834a89d8,5f9555e81729138a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=6ad963b1834a89d8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:33:01.796 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6ad963b1834a89d8,5f9555e81729138a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=6ad963b1834a89d8, duration=2 ms] 2024-10-16 13:33:01.994 INFO 4486 --- [prometheus-minute-schedule-1,73558d8fbddb7cef,2c40d38ae7a42cd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:02.000 INFO 4486 --- [prometheus-minute-schedule-1,73558d8fbddb7cef,2c40d38ae7a42cd5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:02.986 INFO 4486 --- [ocp-async-10,51d5c1dc2d372459,9d4f074531ec6321] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:33:03.039 INFO 4486 --- [ocp-async-9,3e64766d3bdd8451,dc098c4a6df4cbe3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:03.731 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:04.994 INFO 4486 --- [prometheus-minute-schedule-1,0270ec5434756f8d,6fc3395487774cd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:04.999 INFO 4486 --- [prometheus-minute-schedule-1,0270ec5434756f8d,6fc3395487774cd2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:05.039 INFO 4486 --- [ocp-async-2,dad48ba3d74d06d8,e408069399f5951c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:06.732 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:07.036 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,137bf65e8103afa9,424bcc87089b2e15] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=137bf65e8103afa9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:07.037 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,137bf65e8103afa9,424bcc87089b2e15] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=137bf65e8103afa9, duration=2 ms] 2024-10-16 13:33:07.039 INFO 4486 --- [ocp-async-1,5e1b8793dd906e75,21902db59c8e429c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:07.066 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:33:07.066 INFO 4486 --- [ocp-schedules-14,ba64cdb5df8fca8e,8e97398eb03248bc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:33:07.069 INFO 4486 --- [ocp-schedules-14,ba64cdb5df8fca8e,8e97398eb03248bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:07.081 INFO 4486 --- [ocp-schedules-29,8ab08f11335086ff,bcd1c717339bc91a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:33:07.084 INFO 4486 --- [ocp-schedules-29,8ab08f11335086ff,bcd1c717339bc91a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:07.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:33:07.089 INFO 4486 --- [ocp-schedules-17,259bd34d9bfa1c29,44b4e92afc317f3a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:33:07.089 INFO 4486 --- [ocp-schedules-20,217b48e06e81d396,c8579461850351b0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:33:07.090 INFO 4486 --- [ocp-schedules-6,d8980e57dd549162,8df3bea6895c963c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:33:07.090 INFO 4486 --- [host-check-agent-status1,259bd34d9bfa1c29,3d40ca102744d312] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:33:07.090 INFO 4486 --- [host-check-clockdiff1,217b48e06e81d396,09c5bd859f1a6d71] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:33:07.090 INFO 4486 --- [host-check-clockdiff2,217b48e06e81d396,c8a09cfa8d4755be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:33:07.091 INFO 4486 --- [host-check-clockdiff3,217b48e06e81d396,44d698c43910ceb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:33:07.091 INFO 4486 --- [host-check-agent-status2,259bd34d9bfa1c29,3907b8b283975c5f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:33:07.091 INFO 4486 --- [host-check-clockdiff4,217b48e06e81d396,76878bffe372e0e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:33:07.091 INFO 4486 --- [host-check-agent-status3,259bd34d9bfa1c29,5ff869e41e0afc97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:33:07.091 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:33:07.092 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:33:07.092 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:33:07.093 INFO 4486 --- [host-check-agent-status4,259bd34d9bfa1c29,edd9da7bd0dbbd16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:33:07.093 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:33:07.097 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:07.098 INFO 4486 --- [host-check-agent-status1,259bd34d9bfa1c29,3d40ca102744d312] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:07.098 INFO 4486 --- [host-check-agent-status3,259bd34d9bfa1c29,5ff869e41e0afc97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:33:07.099 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:07.099 INFO 4486 --- [host-check-agent-status2,259bd34d9bfa1c29,3907b8b283975c5f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:33:07.100 INFO 4486 --- [host-check-agent-status1,259bd34d9bfa1c29,3d40ca102744d312] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:07.100 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:33:07.101 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:33:07.101 INFO 4486 --- [host-check-agent-status2,259bd34d9bfa1c29,3907b8b283975c5f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:33:07.101 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:33:07.101 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:33:07.102 INFO 4486 --- [host-check-agent-status3,259bd34d9bfa1c29,5ff869e41e0afc97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:33:07.102 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:33:07.103 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:33:07.122 INFO 4486 --- [host-check-agent-status4,259bd34d9bfa1c29,edd9da7bd0dbbd16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:33:07.132 INFO 4486 --- [host-check-agent-status4,259bd34d9bfa1c29,edd9da7bd0dbbd16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:33:07.132 INFO 4486 --- [host-check-clockdiff4,217b48e06e81d396,76878bffe372e0e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056787 3 3 , error= 2024-10-16 13:33:07.132 INFO 4486 --- [host-check-clockdiff1,217b48e06e81d396,09c5bd859f1a6d71] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056787 0 0 , error= 2024-10-16 13:33:07.141 INFO 4486 --- [host-check-clockdiff2,217b48e06e81d396,c8a09cfa8d4755be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056787 3 3 , error= 2024-10-16 13:33:07.146 INFO 4486 --- [host-check-clockdiff3,217b48e06e81d396,44d698c43910ceb0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056787 3 3 , error= 2024-10-16 13:33:07.147 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.149 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:33:07.149 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:33:07.163 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:33:07.164 INFO 4486 --- [host-check-agent-status1,259bd34d9bfa1c29,3d40ca102744d312] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:33:07.164 INFO 4486 --- [host-check-agent-status3,259bd34d9bfa1c29,5ff869e41e0afc97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:33:07.165 INFO 4486 --- [host-check-agent-status2,259bd34d9bfa1c29,3907b8b283975c5f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:33:07.167 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:33:07.167 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:33:07.166779543+08:00 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status2,259bd34d9bfa1c29,3907b8b283975c5f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status2,259bd34d9bfa1c29,3907b8b283975c5f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status2,259bd34d9bfa1c29,3907b8b283975c5f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status3,259bd34d9bfa1c29,5ff869e41e0afc97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status1,259bd34d9bfa1c29,3d40ca102744d312] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status3,259bd34d9bfa1c29,5ff869e41e0afc97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status1,259bd34d9bfa1c29,3d40ca102744d312] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status3,259bd34d9bfa1c29,5ff869e41e0afc97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-agent-status1,259bd34d9bfa1c29,3d40ca102744d312] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:33:07.172 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:33:07.172+08:00 2024-10-16 13:33:07.173 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.173 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:33:07.173 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:33:07.176525767+08:00 2024-10-16 13:33:07.174 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:33:07.173+08:00 2024-10-16 13:33:07.174 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:33:07.177399904+08:00 2024-10-16 13:33:07.174 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:33:07.174 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:33:07.175 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:33:07.174+08:00 2024-10-16 13:33:07.175 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:33:07.172+08:00 and status AVAILABLE 2024-10-16 13:33:07.176 INFO 4486 --- [host-check-agent-status4,259bd34d9bfa1c29,edd9da7bd0dbbd16] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:33:07.180 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:33:07.180 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:33:07.173+08:00 and status AVAILABLE 2024-10-16 13:33:07.180 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:33:07.179567441+08:00 2024-10-16 13:33:07.180 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:33:07.180 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:07.180 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:33:07.174+08:00 and status AVAILABLE 2024-10-16 13:33:07.181 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:33:07.180+08:00 2024-10-16 13:33:07.182 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.182 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:33:07.183 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:33:07.184 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:33:07.180+08:00 and status AVAILABLE 2024-10-16 13:33:07.186 INFO 4486 --- [host-check-agent-status4,259bd34d9bfa1c29,edd9da7bd0dbbd16] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:33:07.186 INFO 4486 --- [host-check-agent-status4,259bd34d9bfa1c29,edd9da7bd0dbbd16] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:33:07.186 INFO 4486 --- [host-check-agent-status4,259bd34d9bfa1c29,edd9da7bd0dbbd16] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:33:07.186 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:33:07.187 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:33:07.190 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:07.191 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:33:07.192 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:33:07.192 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:07.193 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:33:07.193 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.194 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:33:07.194 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:33:07.196 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:33:07.198 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:33:07.203 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.204 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:33:07.214 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.214 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:07.215 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:33:07.216 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:07.218 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.218 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:07.218 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:33:07.223 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:07.228 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:07.230 INFO 4486 --- [ocp-schedules-13,391e0781b60ce92c,9f7370aae80bdc63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:33:07.244 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:33:07.247 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:33:07.258 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:33:07.259 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:33:07.261 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:33:07.267 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:33:07.274 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:33:07.274 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:07.274 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:07.278 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:33:07.290 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:07.294 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:07.305 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:33:07.310 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:33:07.316 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:33:07.321 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:33:07.327 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:33:07.334 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:07.339 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:07.339 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:33:07.354 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:33:07.357 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:07.365 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:33:07.366 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:33:07.368 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:33:07.376 INFO 4486 --- [host-check-host-status3,d8980e57dd549162,6beb428a54e28e60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:33:07.377 INFO 4486 --- [host-check-host-status2,d8980e57dd549162,abdf92d10d1dc036] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:33:07.382 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:07.388 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:33:07.398 INFO 4486 --- [host-check-host-status4,d8980e57dd549162,939260ef595c0831] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:33:07.435 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:33:07.446 INFO 4486 --- [host-check-host-status1,d8980e57dd549162,3c135aae2cf3f022] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:33:07.994 INFO 4486 --- [prometheus-minute-schedule-1,fdce4bfa2ab74582,22b8a6b3bef3e977] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:08.001 INFO 4486 --- [prometheus-minute-schedule-1,fdce4bfa2ab74582,22b8a6b3bef3e977] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:08.907 INFO 4486 --- [ocp-schedules-2,7618b58d11c9a1b2,b1e30fd650b482f1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:33:08.948 WARN 4486 --- [scanner-13247938-1,7618b58d11c9a1b2,bfbec9f6451235ed] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:33:08.981 INFO 4486 --- [alarm-detect-13,7618b58d11c9a1b2,a3aa39a541735cb0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:08.999 INFO 4486 --- [alarm-detect-13,7618b58d11c9a1b2,a3aa39a541735cb0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:09.006 INFO 4486 --- [alarm-detect-13,7618b58d11c9a1b2,a3aa39a541735cb0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:09.016 INFO 4486 --- [alarm-detect-14,7618b58d11c9a1b2,c963f0463add0b31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:09.018 INFO 4486 --- [alarm-detect-19,7618b58d11c9a1b2,5171ebe6d77a055e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:09.022 INFO 4486 --- [alarm-detect-14,7618b58d11c9a1b2,c963f0463add0b31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:09.023 INFO 4486 --- [alarm-detect-19,7618b58d11c9a1b2,5171ebe6d77a055e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:09.026 INFO 4486 --- [alarm-detect-14,7618b58d11c9a1b2,c963f0463add0b31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:09.028 INFO 4486 --- [alarm-detect-19,7618b58d11c9a1b2,5171ebe6d77a055e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:09.041 INFO 4486 --- [ocp-async-10,af45d8e6fc81db51,012891b03c935b98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:09.074 INFO 4486 --- [alarm-detect-13,7618b58d11c9a1b2,cf215739690b8c9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:09.078 INFO 4486 --- [alarm-detect-13,7618b58d11c9a1b2,cf215739690b8c9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:09.083 INFO 4486 --- [alarm-detect-13,7618b58d11c9a1b2,cf215739690b8c9d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:09.261 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b3763edaec52c972,6a2c7e377f0c8640] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b3763edaec52c972, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:09.261 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b3763edaec52c972,6a2c7e377f0c8640] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b3763edaec52c972, duration=1 ms] 2024-10-16 13:33:09.411 INFO 4486 --- [ocp-schedules-15,75107ff3d2e5f63a,324c96fb43079321] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-10-16 13:33:09.413 INFO 4486 --- [ocp-schedules-15,75107ff3d2e5f63a,324c96fb43079321] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:09.449 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:09.452 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:09.454 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:03:03.102283, 2024-10-16 13:33:09.421, 1000] 2024-10-16 13:33:09.456 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-10-16 02:03:03.102283, endTime 2024-10-16 13:33:09.421 2024-10-16 13:33:09.459 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:42:50.879398, 2024-10-16 13:33:09.421, 1000] 2024-10-16 13:33:09.465 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:09.470 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 5, startTime 2024-10-16 08:42:50.879398, endTime 2024-10-16 13:33:09.421 2024-10-16 13:33:09.470 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:06:51.05999, 2024-10-16 13:33:09.421, 1000] 2024-10-16 13:33:09.475 INFO 4486 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:09.476 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000003, startTime 2024-10-16 02:06:51.05999, endTime 2024-10-16 13:33:09.421 2024-10-16 13:33:09.481 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:09.491 INFO 4486 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:09.492 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:09.506 INFO 4486 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:09.508 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:09.511 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:00:14.755425, 2024-10-16 13:33:09.465, 1000] 2024-10-16 13:33:09.512 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-10-16 08:00:14.755425, endTime 2024-10-16 13:33:09.465 2024-10-16 13:33:09.522 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 12:59:45.930954, 2024-10-16 13:33:09.474, 1000] 2024-10-16 13:33:09.526 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 5, startTime 2024-10-16 12:59:45.930954, endTime 2024-10-16 13:33:09.474 2024-10-16 13:33:09.537 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-12 15:26:31.336743, 2024-10-16 13:33:09.479, 1000] 2024-10-16 13:33:09.542 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000003, startTime 2024-10-12 15:26:31.336743, endTime 2024-10-16 13:33:09.479 2024-10-16 13:33:09.733 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:10.007 INFO 4486 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:33:10.009 INFO 4486 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:33:10.028 INFO 4486 --- [Check obproxy2,e118d8ce7f68a062,02be61a52e66a99c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:33:10.028 INFO 4486 --- [Check obproxy2,e118d8ce7f68a062,02be61a52e66a99c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:10.030 INFO 4486 --- [Check obproxy1,a895f5bf15ee42ce,b9f46f576f4a3234] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:33:10.030 INFO 4486 --- [Check obproxy1,a895f5bf15ee42ce,b9f46f576f4a3234] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:10.034 INFO 4486 --- [Collect obproxy1,037d5568bbf3417b,0285a836bd292f4a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:33:10.035 INFO 4486 --- [Collect obproxy1,037d5568bbf3417b,0285a836bd292f4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:10.036 INFO 4486 --- [Collect obproxy2,f7a0b5e85c627b06,8c01486e62fd2982] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:33:10.036 INFO 4486 --- [Collect obproxy2,f7a0b5e85c627b06,8c01486e62fd2982] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:10.050 INFO 4486 --- [Check obproxy2,e118d8ce7f68a062,02be61a52e66a99c] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:33:10.050 INFO 4486 --- [Check obproxy2,e118d8ce7f68a062,02be61a52e66a99c] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:33:10.056 INFO 4486 --- [Collect obproxy1,037d5568bbf3417b,0285a836bd292f4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:33:10.059 INFO 4486 --- [Collect obproxy1,037d5568bbf3417b,0285a836bd292f4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:33:10.060 INFO 4486 --- [Collect obproxy2,f7a0b5e85c627b06,8c01486e62fd2982] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:33:10.061 INFO 4486 --- [Check obproxy1,a895f5bf15ee42ce,b9f46f576f4a3234] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:33:10.061 INFO 4486 --- [Check obproxy1,a895f5bf15ee42ce,b9f46f576f4a3234] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:33:10.062 INFO 4486 --- [Collect obproxy1,037d5568bbf3417b,0285a836bd292f4a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:33:10.065 INFO 4486 --- [Collect obproxy2,f7a0b5e85c627b06,8c01486e62fd2982] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:33:10.070 INFO 4486 --- [Collect obproxy2,f7a0b5e85c627b06,8c01486e62fd2982] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:33:10.280 INFO 4486 --- [ocp-schedules-8,a8cbe8e16573edff,7ca260fefd4cffc0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:33:10.849 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f63f8d8bd47717ad,1ba352456484208c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f63f8d8bd47717ad, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:10.849 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f63f8d8bd47717ad,1ba352456484208c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f63f8d8bd47717ad, duration=2 ms] 2024-10-16 13:33:10.978 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:33:10.994 INFO 4486 --- [prometheus-minute-schedule-1,33e431e8cb2f6263,06e1518a93cfa17c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:11.002 INFO 4486 --- [prometheus-minute-schedule-1,33e431e8cb2f6263,06e1518a93cfa17c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:11.008 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:33:11.041 INFO 4486 --- [ocp-async-8,faf0ec29777c096c,04f092366f643584] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:11.218 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,cddd6e342b1163c6,04c1b75c238cc2a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cddd6e342b1163c6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:11.218 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,cddd6e342b1163c6,04c1b75c238cc2a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cddd6e342b1163c6, duration=2 ms] 2024-10-16 13:33:12.519 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2b03e632090561d0,9b7bbf0f5ba773af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=2b03e632090561d0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:33:12.521 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2b03e632090561d0,9b7bbf0f5ba773af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=2b03e632090561d0, duration=4 ms] 2024-10-16 13:33:12.734 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:12.986 INFO 4486 --- [ocp-async-2,70005eb8a9a662c1,ac79095691605feb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:33:13.041 INFO 4486 --- [ocp-async-3,28111951d99076d7,aa3310a7166a0334] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:13.133 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056720, nodeCount=1000, affectRows=1000 2024-10-16 13:33:13.137 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056720, nodeCount=1000, validCount=1000 2024-10-16 13:33:13.141 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056720, nodeCount=1000, affectRows=1000 2024-10-16 13:33:13.145 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056720, nodeCount=998, validCount=998 2024-10-16 13:33:13.155 ERROR 4486 --- [metric-archive-second-11,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990814) Duplicate entry '669741696473960448-1729056720' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:33:13.153880] [YB420AC531B8-00060B7DB41958EF-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990814) Duplicate entry '669741696473960448-1729056720' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:33:13.153880] [YB420AC531B8-00060B7DB41958EF-0-0] 2024-10-16 13:33:13.194 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056720, nodeCount=1000, affectRows=1000 2024-10-16 13:33:13.197 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056720, nodeCount=1000, validCount=1000 2024-10-16 13:33:13.402 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056720, nodeCount=449, affectRows=449 2024-10-16 13:33:13.403 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056720, nodeCount=448, validCount=448 2024-10-16 13:33:13.452 INFO 4486 --- [ocp-schedules-20,060f418f0cb77523,8d53675367ab3257] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:33:13.454 INFO 4486 --- [ocp-schedules-20,060f418f0cb77523,8d53675367ab3257] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:13.455 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:33:13.455 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:33:13.455 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:33:13.994 INFO 4486 --- [prometheus-minute-schedule-1,553e6e5199e10876,1ea17f15861493e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:14.000 INFO 4486 --- [prometheus-minute-schedule-1,553e6e5199e10876,1ea17f15861493e9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:14.386 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056720, nodeCount=480, affectRows=480 2024-10-16 13:33:15.041 INFO 4486 --- [ocp-async-4,3626edaab2d87a7c,d3f4fe480cd99ce3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:15.100 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056720, nodeCount=480, validCount=480 2024-10-16 13:33:15.734 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:16.994 INFO 4486 --- [prometheus-minute-schedule-1,910ef560239845f0,6af80ebc482ddb63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:17.000 INFO 4486 --- [prometheus-minute-schedule-1,910ef560239845f0,6af80ebc482ddb63] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:17.041 INFO 4486 --- [ocp-async-8,4698b8e7e1198c92,8013ffeae0e5578b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:17.042 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ccdeb5d7a5385734,ef8a518d0bc4bc05] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ccdeb5d7a5385734, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:17.042 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ccdeb5d7a5385734,ef8a518d0bc4bc05] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ccdeb5d7a5385734, duration=1 ms] 2024-10-16 13:33:17.070 INFO 4486 --- [ocp-schedules-14,ba64cdb5df8fca8e,8e97398eb03248bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:33:17.086 INFO 4486 --- [ocp-schedules-27,5a9cd1ac0c79bb88,f4bcc17f636c3154] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:33:17.180 INFO 4486 --- [ocp-schedules-29,8ab08f11335086ff,bcd1c717339bc91a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:33:18.600 INFO 4486 --- [ocp-schedules-17,234267c01187ff98,52ebcf54ddfa593a] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:33:18.601 INFO 4486 --- [ocp-schedules-17,234267c01187ff98,52ebcf54ddfa593a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:18.735 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:18.853 INFO 4486 --- [ocp-schedules-26,3b73c5c8ccb6794a,97e67114d9a3f636] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:33:19.042 INFO 4486 --- [ocp-async-7,7f175c98da8b3a38,f1687313d97801bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:19.102 INFO 4486 --- [ocp-schedules-27,58a94dd0692949c7,332762dd0e1b140f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:33:19.126 WARN 4486 --- [scanner-13249073-1,58a94dd0692949c7,33f26101a038909f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:33:19.186 INFO 4486 --- [alarm-detect-17,58a94dd0692949c7,b9057ea30e2ed30f] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056765 ~ 1729056789 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056780, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056780, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056780=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:33:19.266 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,91683eaec7c1757d,a0ce93815f1fd92c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=91683eaec7c1757d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:19.266 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,91683eaec7c1757d,a0ce93815f1fd92c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=91683eaec7c1757d, duration=1 ms] 2024-10-16 13:33:19.543 INFO 4486 --- [ocp-schedules-15,75107ff3d2e5f63a,324c96fb43079321] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=3, successCount=3 2024-10-16 13:33:19.994 INFO 4486 --- [prometheus-minute-schedule-1,8d61429a4ef5fdfa,0f1fe07e26a8432b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:20.000 INFO 4486 --- [prometheus-minute-schedule-1,8d61429a4ef5fdfa,0f1fe07e26a8432b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:20.286 INFO 4486 --- [ocp-schedules-1,9a5cb92bb43c532e,04ec6ea93dca157b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:33:20.855 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,700a6d2638bb134b,232ef6a01267798b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=700a6d2638bb134b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:20.855 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,700a6d2638bb134b,232ef6a01267798b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=700a6d2638bb134b, duration=1 ms] 2024-10-16 13:33:20.996 INFO 4486 --- [prometheus-cache-monitor1,e7f8bba1cd425372,453e61e15d606b85] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:33:20.996 INFO 4486 --- [prometheus-cache-monitor1,e7f8bba1cd425372,453e61e15d606b85] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:33:21.042 INFO 4486 --- [ocp-async-1,6c7b09cfb4d1452f,90eae02c53feeca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:21.223 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7f25ab8292daed15,263690950caac8d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7f25ab8292daed15, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:21.223 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7f25ab8292daed15,263690950caac8d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7f25ab8292daed15, duration=1 ms] 2024-10-16 13:33:21.736 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:22.733 INFO 4486 --- [ocp-schedules-18,5f425fe1dbfdb30b,ddf9b444ac3a17f9] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:33:22.987 INFO 4486 --- [ocp-async-4,123f91d425ddc24f,091275bf5826e16e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:33:22.994 INFO 4486 --- [prometheus-minute-schedule-1,48156799155891d6,cc2c7cb6aad7beeb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:22.999 INFO 4486 --- [prometheus-minute-schedule-1,48156799155891d6,cc2c7cb6aad7beeb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:23.042 INFO 4486 --- [ocp-async-8,322dd196c1aae4e0,6f38f31c9ff4e462] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:23.455 INFO 4486 --- [ocp-schedules-20,060f418f0cb77523,8d53675367ab3257] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:33:24.083 INFO 4486 --- [ocp-schedules-30,c25634d9e2ca00b6,c0754b71c8e67c8f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:33:24.171 INFO 4486 --- [ocp-schedules-11,ab6c9625e71daa85,32934e3db085b751] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:33:24.450 INFO 4486 --- [ocp-schedules-7,f159acfeeccae4dc,c56e1ab127bc828f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:33:24.450 INFO 4486 --- [ocp-schedules-7,f159acfeeccae4dc,c56e1ab127bc828f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:33:24.483 INFO 4486 --- [Collect all tenants major compactions2,f159acfeeccae4dc,a9c85e263b0bd4a9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:33:24.483 INFO 4486 --- [Collect all tenants major compactions3,f159acfeeccae4dc,72d6b65f4d7abf5a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:33:24.484 INFO 4486 --- [Collect all tenants major compactions1,f159acfeeccae4dc,afe68408a6b93d15] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:33:24.484 INFO 4486 --- [Collect all tenants major compactions4,f159acfeeccae4dc,332fe52eb77f30fa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:33:24.485 INFO 4486 --- [Collect all tenants major compactions5,f159acfeeccae4dc,dcbf0703d8756c1b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:33:24.497 INFO 4486 --- [Collect all tenants major compactions2,f159acfeeccae4dc,a9c85e263b0bd4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:24.498 INFO 4486 --- [Collect all tenants major compactions1,f159acfeeccae4dc,afe68408a6b93d15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:24.499 INFO 4486 --- [Collect all tenants major compactions1,f159acfeeccae4dc,afe68408a6b93d15] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:33:24.499 INFO 4486 --- [Collect all tenants major compactions2,f159acfeeccae4dc,a9c85e263b0bd4a9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:33:24.502 INFO 4486 --- [Collect all tenants major compactions2,f159acfeeccae4dc,a9c85e263b0bd4a9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:33:24.503 INFO 4486 --- [Collect all tenants major compactions1,f159acfeeccae4dc,afe68408a6b93d15] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:33:24.503 INFO 4486 --- [Collect all tenants major compactions2,f159acfeeccae4dc,a9c85e263b0bd4a9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:33:24.503 INFO 4486 --- [Collect all tenants major compactions3,f159acfeeccae4dc,72d6b65f4d7abf5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:24.503 INFO 4486 --- [Collect all tenants major compactions1,f159acfeeccae4dc,afe68408a6b93d15] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:33:24.504 INFO 4486 --- [Collect all tenants major compactions3,f159acfeeccae4dc,72d6b65f4d7abf5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:33:24.508 INFO 4486 --- [Collect all tenants major compactions3,f159acfeeccae4dc,72d6b65f4d7abf5a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:33:24.508 INFO 4486 --- [Collect all tenants major compactions3,f159acfeeccae4dc,72d6b65f4d7abf5a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:33:24.515 INFO 4486 --- [Collect all tenants major compactions4,f159acfeeccae4dc,332fe52eb77f30fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:24.515 INFO 4486 --- [Collect all tenants major compactions5,f159acfeeccae4dc,dcbf0703d8756c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:24.516 INFO 4486 --- [Collect all tenants major compactions4,f159acfeeccae4dc,332fe52eb77f30fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:33:24.517 INFO 4486 --- [Collect all tenants major compactions5,f159acfeeccae4dc,dcbf0703d8756c1b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:33:24.519 INFO 4486 --- [Collect all tenants major compactions4,f159acfeeccae4dc,332fe52eb77f30fa] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:33:24.519 INFO 4486 --- [Collect all tenants major compactions4,f159acfeeccae4dc,332fe52eb77f30fa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:33:24.522 INFO 4486 --- [Collect all tenants major compactions5,f159acfeeccae4dc,dcbf0703d8756c1b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:33:24.522 INFO 4486 --- [Collect all tenants major compactions5,f159acfeeccae4dc,dcbf0703d8756c1b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:33:24.523 INFO 4486 --- [ocp-schedules-7,f159acfeeccae4dc,c56e1ab127bc828f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:33:24.737 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:25.042 INFO 4486 --- [ocp-async-7,d2c6da69448cd30b,801305326fe0d388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:25.994 INFO 4486 --- [prometheus-minute-schedule-1,e6192397a203e381,b224fbdcbccb688a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:26.000 INFO 4486 --- [prometheus-minute-schedule-1,e6192397a203e381,b224fbdcbccb688a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:26.861 INFO 4486 --- [ocp-schedules-21,138d9db736940cfe,89640b3cf152b26f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:33:26.861+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:33:26.863 INFO 4486 --- [ocp-schedules-21,138d9db736940cfe,89640b3cf152b26f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:33:26.864 INFO 4486 --- [ocp-schedules-21,138d9db736940cfe,89640b3cf152b26f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:33:26.865 INFO 4486 --- [ocp-schedules-21,138d9db736940cfe,89640b3cf152b26f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:33:27.042 INFO 4486 --- [ocp-async-1,a832953465fd50db,17e7a3c06ddeb980] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:27.048 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,978bb373623f5d3c,1420b80e60b1846d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=978bb373623f5d3c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:27.048 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,978bb373623f5d3c,1420b80e60b1846d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=978bb373623f5d3c, duration=3 ms] 2024-10-16 13:33:27.738 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:27.799 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,fbe57c9e70e9c468,855e346158313a30] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=fbe57c9e70e9c468, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:33:27.800 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,fbe57c9e70e9c468,855e346158313a30] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=fbe57c9e70e9c468, duration=2 ms] 2024-10-16 13:33:28.630 INFO 4486 --- [ocp-schedules-17,234267c01187ff98,52ebcf54ddfa593a] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:33:28.994 INFO 4486 --- [prometheus-minute-schedule-1,f586ef8fd4866645,ddd53b3bf3d8c8dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:29.001 INFO 4486 --- [prometheus-minute-schedule-1,f586ef8fd4866645,ddd53b3bf3d8c8dc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:29.043 INFO 4486 --- [ocp-async-10,008323934a2678d4,04d6e62b83036153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:29.237 INFO 4486 --- [ocp-schedules-12,78190c4866b9cf42,b1592df721881a77] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:33:29.283 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d44ac4f40ab0ca00,517933e77fd6621e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d44ac4f40ab0ca00, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:29.284 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d44ac4f40ab0ca00,517933e77fd6621e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d44ac4f40ab0ca00, duration=8 ms] 2024-10-16 13:33:29.303 WARN 4486 --- [scanner-13249333-1,78190c4866b9cf42,f8ba02734fa8561b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:33:29.313 INFO 4486 --- [alarm-detect-14,78190c4866b9cf42,1d9b2df0703ec374] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:29.319 INFO 4486 --- [alarm-detect-14,78190c4866b9cf42,1d9b2df0703ec374] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:29.328 INFO 4486 --- [alarm-detect-14,78190c4866b9cf42,1d9b2df0703ec374] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:29.332 INFO 4486 --- [alarm-detect-19,78190c4866b9cf42,ba30eb5153084282] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:29.333 INFO 4486 --- [alarm-detect-9,78190c4866b9cf42,4a183612c4c2acee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:29.336 INFO 4486 --- [alarm-detect-19,78190c4866b9cf42,ba30eb5153084282] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:29.339 INFO 4486 --- [alarm-detect-9,78190c4866b9cf42,4a183612c4c2acee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:29.340 INFO 4486 --- [alarm-detect-19,78190c4866b9cf42,ba30eb5153084282] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:29.343 INFO 4486 --- [alarm-detect-9,78190c4866b9cf42,4a183612c4c2acee] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:29.359 INFO 4486 --- [alarm-detect-6,78190c4866b9cf42,6cd2e8e93733cb53] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056780 ~ 1729056799 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056795, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056795, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056795=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:33:29.466 INFO 4486 --- [alarm-detect-13,78190c4866b9cf42,0a1ae2b2cf1c644e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:29.475 INFO 4486 --- [alarm-detect-13,78190c4866b9cf42,0a1ae2b2cf1c644e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:29.483 INFO 4486 --- [alarm-detect-13,78190c4866b9cf42,0a1ae2b2cf1c644e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:30.103 INFO 4486 --- [prometheus-second-schedule-1,108c1db793362cf2,b4c8c914b453e7e4] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:33:30.293 INFO 4486 --- [ocp-schedules-26,585563a09875d748,2e36b3b3317340af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:33:30.421 INFO 4486 --- [ocp-schedules-16,bc12cc82494ea965,c56479a7a5b4325b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:33:30.421 INFO 4486 --- [ocp-schedules-16,296f6bcb6168df90,dab8de754f6d98ae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:33:30.424 INFO 4486 --- [ocp-schedules-16,296f6bcb6168df90,dab8de754f6d98ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:30.453 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.454 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:33:30.463 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.464 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:33:30.489 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:30.490 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.491 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:33:30.491 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.492 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:33:30.527 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:30.528 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.529 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:33:30.530 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:33:30.530 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.535 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:33:30.537 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:33:30.548 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.549 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:33:30.562 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.573 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:30.574 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.574 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:33:30.575 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:33:30.583 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:33:30.590 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:30.593 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:30.613 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:30.615 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.616 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:33:30.619 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:33:30.645 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:33:30.645 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:33:30.656 WARN 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : Failed to get ob process info, errMsg:[AgentClient]:http request is failed, response:Cannot get process info of open /proc/18759/status: no such file or directory, reason: %!v(MISSING) 2024-10-16 13:33:30.668 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:30.669 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.670 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:33:30.673 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:30.674 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.675 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:33:30.680 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:33:30.680 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:33:30.680 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.683 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:33:30.687 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:33:30.715 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.716 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:33:30.738 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:30.743 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.744 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:33:30.771 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.771 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:33:30.803 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.803 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:33:30.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:33:30.860 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2b196696c363793b,f250c3021e8d9a33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2b196696c363793b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:30.860 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2b196696c363793b,f250c3021e8d9a33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2b196696c363793b, duration=2 ms] 2024-10-16 13:33:30.861 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:33:30.888 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:30.890 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:30.891 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:33:30.895 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:33:30.899 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:33:30.903 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:33:30.905 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:33:30.930 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:33:31.043 INFO 4486 --- [ocp-async-2,4eae55c686d5efea,6194bcad70a1afa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:31.090 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:33:31.118 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:31.120 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:31.121 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:33:31.153 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:33:31.154 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056228392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:33:31.228 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,26e45d0a610fe558,dbef8b221cdc0012] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=26e45d0a610fe558, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:31.228 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,26e45d0a610fe558,dbef8b221cdc0012] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=26e45d0a610fe558, duration=1 ms] 2024-10-16 13:33:31.994 INFO 4486 --- [prometheus-minute-schedule-1,35bcb7208fb89b65,db885c0d80541d36] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:32.000 INFO 4486 --- [prometheus-minute-schedule-1,35bcb7208fb89b65,db885c0d80541d36] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:32.987 INFO 4486 --- [ocp-async-7,f3359d9219ecc132,3e55d54a380576a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:33:33.043 INFO 4486 --- [ocp-async-6,14f591b3869f7146,9c11729b26457f00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:33.739 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:34.994 INFO 4486 --- [prometheus-minute-schedule-1,7b108aecf809242b,fa8c5777f4aae31f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:35.001 INFO 4486 --- [prometheus-minute-schedule-1,7b108aecf809242b,fa8c5777f4aae31f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:35.043 INFO 4486 --- [ocp-async-9,446e6d161536be56,d6af76677acb3fd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:36.740 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:37.043 INFO 4486 --- [ocp-async-5,b530b91845facd0e,214407598eefdaf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:37.058 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,13993b382ac52bb1,bae69ca8af654f3e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=13993b382ac52bb1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:37.058 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,13993b382ac52bb1,bae69ca8af654f3e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=13993b382ac52bb1, duration=2 ms] 2024-10-16 13:33:37.063 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:33:37.064 INFO 4486 --- [ocp-schedules-8,bd6adf27a5b4cb5c,7acda1ebf1a61383] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:33:37.066 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:33:37.067 INFO 4486 --- [ocp-schedules-26,047a86d9f83a9394,59184a0d8f2493ae] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:33:37.067 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:33:37.068 INFO 4486 --- [ocp-schedules-26,047a86d9f83a9394,59184a0d8f2493ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:37.069 INFO 4486 --- [ocp-schedules-8,bd6adf27a5b4cb5c,7acda1ebf1a61383] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:33:37.082 INFO 4486 --- [ocp-schedules-27,7ee1dfdded1cc4fc,ebb7f6290494bf31] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:33:37.084 INFO 4486 --- [ocp-schedules-27,7ee1dfdded1cc4fc,ebb7f6290494bf31] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:33:37.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:33:37.089 INFO 4486 --- [ocp-schedules-9,fa97ffb7f7fefba6,7e8be5ec0e0563ed] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:33:37.089 INFO 4486 --- [ocp-schedules-23,c4f979e6de04bfd7,3ba7966fbb0bb977] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:33:37.090 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:33:37.090 INFO 4486 --- [host-check-clockdiff1,c4f979e6de04bfd7,ab57bcdc23ad2580] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:33:37.090 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:33:37.090 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:33:37.091 INFO 4486 --- [host-check-clockdiff3,c4f979e6de04bfd7,81d658f3bf78f42b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:33:37.091 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:33:37.091 INFO 4486 --- [host-check-clockdiff2,c4f979e6de04bfd7,e7335ad5e5f76e9d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:33:37.091 INFO 4486 --- [host-check-clockdiff4,c4f979e6de04bfd7,5e57289a648d6068] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:33:37.101 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:33:37.101 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:37.101 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:33:37.117 INFO 4486 --- [host-check-clockdiff1,c4f979e6de04bfd7,ab57bcdc23ad2580] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056817 0 0 , error= 2024-10-16 13:33:37.125 INFO 4486 --- [host-check-clockdiff3,c4f979e6de04bfd7,81d658f3bf78f42b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056817 3 3 , error= 2024-10-16 13:33:37.134 INFO 4486 --- [ocp-schedules-4,b844492696d85a6b,5815bd1c321895ad] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:33:37.136 INFO 4486 --- [host-check-clockdiff4,c4f979e6de04bfd7,5e57289a648d6068] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056817 3 3 , error= 2024-10-16 13:33:37.136 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:37.136 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:33:37.137 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:33:37.137 INFO 4486 --- [host-check-agent-status1,b844492696d85a6b,9fd8625097d9fd5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:33:37.137 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:33:37.138 INFO 4486 --- [host-check-agent-status3,b844492696d85a6b,33221e81eaf597c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:33:37.138 INFO 4486 --- [host-check-agent-status2,b844492696d85a6b,2839a90d6d9c495c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:33:37.138 INFO 4486 --- [host-check-agent-status4,b844492696d85a6b,314c80b40e33abd0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:33:37.139 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:33:37.143 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.143 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:33:37.144 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:33:37.144 INFO 4486 --- [host-check-agent-status3,b844492696d85a6b,33221e81eaf597c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:33:37.145 INFO 4486 --- [host-check-agent-status2,b844492696d85a6b,2839a90d6d9c495c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:33:37.146 INFO 4486 --- [host-check-agent-status4,b844492696d85a6b,314c80b40e33abd0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:33:37.146 INFO 4486 --- [host-check-agent-status3,b844492696d85a6b,33221e81eaf597c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:33:37.146 INFO 4486 --- [host-check-agent-status1,b844492696d85a6b,9fd8625097d9fd5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:37.147 INFO 4486 --- [host-check-agent-status2,b844492696d85a6b,2839a90d6d9c495c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:33:37.148 INFO 4486 --- [host-check-agent-status4,b844492696d85a6b,314c80b40e33abd0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:33:37.148 INFO 4486 --- [host-check-agent-status1,b844492696d85a6b,9fd8625097d9fd5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:37.165 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:33:37.165 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:33:37.165 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:33:37.168 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:33:37.170783578+08:00 2024-10-16 13:33:37.168 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:33:37.170627869+08:00 2024-10-16 13:33:37.168 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:33:37.167516938+08:00 2024-10-16 13:33:37.168 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:33:37.168+08:00 2024-10-16 13:33:37.168 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:33:37.168+08:00 2024-10-16 13:33:37.168 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:33:37.168+08:00 2024-10-16 13:33:37.169 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:33:37.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:33:37.171 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:33:37.168+08:00 and status AVAILABLE 2024-10-16 13:33:37.171 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:33:37.168+08:00 and status AVAILABLE 2024-10-16 13:33:37.171 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:33:37.168+08:00 and status AVAILABLE 2024-10-16 13:33:37.172 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:33:37.174616325+08:00 2024-10-16 13:33:37.172 INFO 4486 --- [host-check-agent-status4,b844492696d85a6b,314c80b40e33abd0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:33:37.172 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:33:37.172+08:00 2024-10-16 13:33:37.173 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.173 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:33:37.173 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:33:37.173 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:33:37.174 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:33:37.175 INFO 4486 --- [host-check-agent-status1,b844492696d85a6b,9fd8625097d9fd5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:33:37.175 INFO 4486 --- [host-check-agent-status2,b844492696d85a6b,2839a90d6d9c495c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:33:37.176 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.176 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:33:37.172+08:00 and status AVAILABLE 2024-10-16 13:33:37.177 INFO 4486 --- [host-check-agent-status4,b844492696d85a6b,314c80b40e33abd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:33:37.177 INFO 4486 --- [host-check-agent-status4,b844492696d85a6b,314c80b40e33abd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:33:37.177 INFO 4486 --- [host-check-agent-status4,b844492696d85a6b,314c80b40e33abd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:33:37.177 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:33:37.178 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:33:37.178 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.179 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.179 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:33:37.179 INFO 4486 --- [host-check-agent-status3,b844492696d85a6b,33221e81eaf597c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:33:37.180 INFO 4486 --- [host-check-agent-status1,b844492696d85a6b,9fd8625097d9fd5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:33:37.180 INFO 4486 --- [host-check-agent-status1,b844492696d85a6b,9fd8625097d9fd5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:33:37.180 INFO 4486 --- [host-check-agent-status1,b844492696d85a6b,9fd8625097d9fd5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:33:37.180 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:33:37.181 INFO 4486 --- [host-check-agent-status2,b844492696d85a6b,2839a90d6d9c495c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:33:37.181 INFO 4486 --- [host-check-agent-status2,b844492696d85a6b,2839a90d6d9c495c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:33:37.181 INFO 4486 --- [host-check-agent-status2,b844492696d85a6b,2839a90d6d9c495c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:33:37.181 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:33:37.181 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:33:37.182 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:37.183 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:33:37.183 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:33:37.183 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:37.183 INFO 4486 --- [host-check-agent-status3,b844492696d85a6b,33221e81eaf597c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:33:37.183 INFO 4486 --- [host-check-agent-status3,b844492696d85a6b,33221e81eaf597c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:33:37.183 INFO 4486 --- [host-check-agent-status3,b844492696d85a6b,33221e81eaf597c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:33:37.186 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.187 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:33:37.187 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:33:37.189 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:33:37.197 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.198 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:33:37.198 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.198 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.198 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.198 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.198 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.199 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.204 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:37.205 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:37.208 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.209 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:33:37.210 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:37.211 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.212 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:33:37.212 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:33:37.216 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:33:37.226 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.227 INFO 4486 --- [ocp-schedules-12,f4c7a1fb1f772c61,3b9e0f3fbedd0ef3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:33:37.234 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:33:37.236 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:33:37.245 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:33:37.245 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:33:37.253 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:33:37.259 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:37.260 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:37.264 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:33:37.271 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:33:37.280 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:37.283 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:33:37.287 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:33:37.292 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:33:37.298 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:33:37.300 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:33:37.302 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:33:37.315 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:37.316 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:37.324 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:33:37.342 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:33:37.353 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:33:37.355 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:33:37.358 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:37.359 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:33:37.363 INFO 4486 --- [host-check-host-status3,fa97ffb7f7fefba6,21362bf8f97d29a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:33:37.367 INFO 4486 --- [host-check-host-status4,fa97ffb7f7fefba6,76cf8df5af0cea72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:33:37.370 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:33:37.388 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:33:37.397 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:33:37.408 INFO 4486 --- [host-check-host-status2,fa97ffb7f7fefba6,c71f4cf30d7bdc52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:33:37.432 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:37.434 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.435 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:33:37.437 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.438 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.438 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:33:37.447 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.447 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.447 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.447 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.447 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.447 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.452 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:33:37.462 INFO 4486 --- [host-check-host-status1,fa97ffb7f7fefba6,85c255ceb0484c91] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:33:37.474 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:33:37.528 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:33:37.575 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:37.593 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:37.595 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.600 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:33:37.601 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:33:37.602 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.603 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:33:37.604 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.605 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.606 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:33:37.616 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.616 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.616 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.616 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.616 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.617 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.625 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:33:37.837 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:33:37.882 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:37.884 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.885 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:33:37.886 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.886 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.887 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:33:37.896 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.896 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.896 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.896 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.896 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:37.897 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:37.914 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:33:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:33:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:33:37.994 INFO 4486 --- [prometheus-minute-schedule-1,ab02e743ccc3b549,71eb4da371233329] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:38.001 INFO 4486 --- [prometheus-minute-schedule-1,ab02e743ccc3b549,71eb4da371233329] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:38.052 INFO 4486 --- [host-check-clockdiff2,c4f979e6de04bfd7,e7335ad5e5f76e9d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056818 3 3 , error= 2024-10-16 13:33:38.131 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:33:38.176 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:33:38.178 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:38.179 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:33:38.180 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:38.181 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:38.182 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:33:38.195 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:38.195 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:38.195 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:38.195 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:38.195 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:33:38.196 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:38.265 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:33:38.466 INFO 4486 --- [ocp-schedules-13,88794b2d6ffd7a3e,c33e2c5da509a22d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:33:38.466 INFO 4486 --- [ocp-schedules-13,88794b2d6ffd7a3e,c33e2c5da509a22d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:33:38.487 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:33:38.269737, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:33:38.487 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:33:38.269737, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:33:38.531 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:33:38.269737} } 2024-10-16 13:33:38.591 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:33:38.598 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:33:38.605 INFO 4486 --- [ocp-schedules-11,876447a6bdeafd17,234167d0c3bdd232] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:33:38.858 INFO 4486 --- [ocp-schedules-6,bd66282f8fac184e,dcf34a7de2dbf252] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:33:39.044 INFO 4486 --- [ocp-async-6,eafa50e60b75a69c,83a83aaa6056b75a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:39.291 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4c72cf3734719a9c,3f67b641483b617a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4c72cf3734719a9c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:39.291 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4c72cf3734719a9c,3f67b641483b617a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4c72cf3734719a9c, duration=2 ms] 2024-10-16 13:33:39.511 INFO 4486 --- [ocp-schedules-30,3b73395215a189dd,1d404433812e7404] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:33:39.572 WARN 4486 --- [scanner-13249677-1,3b73395215a189dd,bf016be0d3b1ada9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:33:39.741 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:40.006 INFO 4486 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:33:40.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:33:40.026 INFO 4486 --- [Check obproxy2,ffc1d48a30ed1bfc,b7ef0ec1474a7b50] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:33:40.026 INFO 4486 --- [Check obproxy1,a10400a877e9b318,33c2ce3431da63c9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:33:40.026 INFO 4486 --- [Check obproxy1,a10400a877e9b318,33c2ce3431da63c9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:40.026 INFO 4486 --- [Check obproxy2,ffc1d48a30ed1bfc,b7ef0ec1474a7b50] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:40.030 INFO 4486 --- [Collect obproxy1,c12a64e3e2e751e9,71772eab9a2db967] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:33:40.030 INFO 4486 --- [Collect obproxy1,c12a64e3e2e751e9,71772eab9a2db967] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:40.031 INFO 4486 --- [Collect obproxy2,3c21ec66d9ce8b0e,b39ef847d5ba3692] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:33:40.031 INFO 4486 --- [Collect obproxy2,3c21ec66d9ce8b0e,b39ef847d5ba3692] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:33:40.051 INFO 4486 --- [Check obproxy2,ffc1d48a30ed1bfc,b7ef0ec1474a7b50] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:33:40.051 INFO 4486 --- [Check obproxy2,ffc1d48a30ed1bfc,b7ef0ec1474a7b50] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:33:40.051 INFO 4486 --- [Check obproxy1,a10400a877e9b318,33c2ce3431da63c9] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:33:40.051 INFO 4486 --- [Check obproxy1,a10400a877e9b318,33c2ce3431da63c9] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:33:40.053 INFO 4486 --- [Collect obproxy2,3c21ec66d9ce8b0e,b39ef847d5ba3692] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:33:40.057 INFO 4486 --- [Collect obproxy2,3c21ec66d9ce8b0e,b39ef847d5ba3692] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:33:40.060 INFO 4486 --- [Collect obproxy1,c12a64e3e2e751e9,71772eab9a2db967] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:33:40.062 INFO 4486 --- [Collect obproxy2,3c21ec66d9ce8b0e,b39ef847d5ba3692] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:33:40.063 INFO 4486 --- [Collect obproxy1,c12a64e3e2e751e9,71772eab9a2db967] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:33:40.066 INFO 4486 --- [Collect obproxy1,c12a64e3e2e751e9,71772eab9a2db967] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:33:40.299 INFO 4486 --- [ocp-schedules-3,54b6633f95b30be8,577548c41302da64] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:33:40.866 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,424453399111d664,d518974f3c02b4d5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=424453399111d664, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:40.866 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,424453399111d664,d518974f3c02b4d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=424453399111d664, duration=2 ms] 2024-10-16 13:33:40.994 INFO 4486 --- [prometheus-minute-schedule-1,d8f6a5cdfe382dd9,7246bd0df5f9d57a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:40.995 INFO 4486 --- [prometheus-hour-schedule-1,b835ad9a21d81eba,fc349f25be3c66d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:33:41.000 INFO 4486 --- [prometheus-minute-schedule-1,d8f6a5cdfe382dd9,7246bd0df5f9d57a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:41.044 INFO 4486 --- [ocp-async-10,60e9443f6b896f99,f3cf833d71262e79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:41.233 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d6c2f2d073192949,2bc895c44426b913] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d6c2f2d073192949, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:41.233 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d6c2f2d073192949,2bc895c44426b913] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d6c2f2d073192949, duration=1 ms] 2024-10-16 13:33:42.083 INFO 4486 --- [ocp-schedules-3,e0bf56326e6b196d,1f92670081f6143a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:33:42.742 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:42.987 INFO 4486 --- [ocp-async-2,9e6f14ba4266c8e1,4e905970fff7ef98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:33:43.044 INFO 4486 --- [ocp-async-7,57f0b68bc7eb8145,bc415e3e605f41e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:43.994 INFO 4486 --- [prometheus-minute-schedule-1,ba7fae99fd6e4c89,93465b6733d3a647] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:44.003 INFO 4486 --- [prometheus-minute-schedule-1,ba7fae99fd6e4c89,93465b6733d3a647] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:45.044 INFO 4486 --- [ocp-async-4,2f804ab1c54f5796,0a0753762288b508] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:45.442 INFO 4486 --- [ocp-schedules-9,fdd77003b5bf44da,83960c07ba815d0e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:33:45.449 INFO 4486 --- [ocp-schedules-9,fdd77003b5bf44da,83960c07ba815d0e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:33:45.449 INFO 4486 --- [ocp-schedules-9,fdd77003b5bf44da,83960c07ba815d0e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:33:45.463 INFO 4486 --- [ocp-schedules-13,651e0f785b6b6de2,8f6fe9c7c2a16403] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:33:45.463 INFO 4486 --- [ocp-schedules-13,651e0f785b6b6de2,8f6fe9c7c2a16403] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:33:45.566 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,eddb6f4df8d7ca8a,f44fd2fe0948ed80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=eddb6f4df8d7ca8a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:33:45.567 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,eddb6f4df8d7ca8a,f44fd2fe0948ed80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=eddb6f4df8d7ca8a, duration=3 ms] 2024-10-16 13:33:45.743 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:46.994 INFO 4486 --- [prometheus-minute-schedule-1,3f673a29f688b720,f22e789cb1ee672e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:47.000 INFO 4486 --- [prometheus-minute-schedule-1,3f673a29f688b720,f22e789cb1ee672e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:47.045 INFO 4486 --- [ocp-async-9,13de7431c4f86092,e2807931e6f850ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:47.063 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4d08399da366c752,7d39c1ea1a0949bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4d08399da366c752, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:47.063 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4d08399da366c752,7d39c1ea1a0949bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4d08399da366c752, duration=1 ms] 2024-10-16 13:33:47.070 INFO 4486 --- [ocp-schedules-26,047a86d9f83a9394,59184a0d8f2493ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:33:47.089 INFO 4486 --- [ocp-schedules-15,ea2952c4ea0bcfa7,cf89cd6492d83d0f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:33:47.175 INFO 4486 --- [ocp-schedules-27,7ee1dfdded1cc4fc,ebb7f6290494bf31] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:33:48.744 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:48.781 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:48.782 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:33:48.783 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:33:48.784 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:33:48.784 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:33:49.045 INFO 4486 --- [ocp-async-2,3534b1c1b91ae756,c393cb645b5f7c19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:49.295 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d2c104257b7d7bfb,a8a25a86cd844922] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d2c104257b7d7bfb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:49.295 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d2c104257b7d7bfb,a8a25a86cd844922] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d2c104257b7d7bfb, duration=1 ms] 2024-10-16 13:33:49.766 INFO 4486 --- [ocp-schedules-2,ad55defc1418f5d4,73f8ad82c1e9c202] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:33:49.813 WARN 4486 --- [scanner-13249900-1,ad55defc1418f5d4,af46afaffa7688ac] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:33:49.836 INFO 4486 --- [alarm-detect-5,ad55defc1418f5d4,cafc19971bd83f25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:49.842 INFO 4486 --- [alarm-detect-5,ad55defc1418f5d4,cafc19971bd83f25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:49.844 INFO 4486 --- [alarm-detect-15,ad55defc1418f5d4,34e735b1d88e6134] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:49.846 INFO 4486 --- [alarm-detect-13,ad55defc1418f5d4,ab9e3fe1b16b468f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:49.846 INFO 4486 --- [alarm-detect-5,ad55defc1418f5d4,cafc19971bd83f25] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 2004835, data is empty. 2024-10-16 13:33:49.849 INFO 4486 --- [alarm-detect-15,ad55defc1418f5d4,34e735b1d88e6134] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:49.851 INFO 4486 --- [alarm-detect-13,ad55defc1418f5d4,ab9e3fe1b16b468f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:49.853 INFO 4486 --- [alarm-detect-15,ad55defc1418f5d4,34e735b1d88e6134] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 2004835, data is empty. 2024-10-16 13:33:49.857 INFO 4486 --- [alarm-detect-13,ad55defc1418f5d4,ab9e3fe1b16b468f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 2004835, data is empty. 2024-10-16 13:33:49.906 INFO 4486 --- [alarm-detect-17,ad55defc1418f5d4,5e080b7e0128b268] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056795 ~ 1729056819 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056810, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056810, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056810=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:33:49.921 INFO 4486 --- [ocp-schedules-18,cbfba46ed1145073,957aed40e4abbf22] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:33:49.926 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:33:49.926 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:33:49.926 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:33:49.926 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:33:49.926 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:33:49.926 INFO 4486 --- [ocp-schedules-18,cbfba46ed1145073,957aed40e4abbf22] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:33:49.947 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:33:49.947 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:33:49.947 INFO 4486 --- [ocp-schedules-18,cbfba46ed1145073,957aed40e4abbf22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:33:49.948 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:33:49.948 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:33:49.948 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:33:49.962 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:33:49.963 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:33:49.962 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:33:49.962 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:33:49.963 INFO 4486 --- [ocp-schedules-18,cbfba46ed1145073,957aed40e4abbf22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:33:49.962 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:33:49.963 INFO 4486 --- [alarm-detect-19,ad55defc1418f5d4,58bde87b442a2b39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:49.969 INFO 4486 --- [alarm-detect-19,ad55defc1418f5d4,58bde87b442a2b39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:49.977 INFO 4486 --- [alarm-detect-19,ad55defc1418f5d4,58bde87b442a2b39] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 2004835, data is empty. 2024-10-16 13:33:49.994 INFO 4486 --- [prometheus-minute-schedule-1,fb9adfd0ca831401,0e1b77d2f23b4b26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:50.002 INFO 4486 --- [prometheus-minute-schedule-1,fb9adfd0ca831401,0e1b77d2f23b4b26] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:50.006 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:33:50.006 INFO 4486 --- [ocp-schedules-18,cbfba46ed1145073,957aed40e4abbf22] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:33:50.006 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:33:50.006 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:33:50.006 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:33:50.007 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:33:50.019 INFO 4486 --- [ocp-schedules-18,cbfba46ed1145073,957aed40e4abbf22] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:33:50.305 INFO 4486 --- [ocp-schedules-30,5c9f5245882501a4,3059a0dc44967fe1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:33:50.373 INFO 4486 --- [ocp-schedules-17,10a6559d47afa512,4aed53f2a54688ee] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:33:50.374 INFO 4486 --- [ocp-schedules-17,10a6559d47afa512,4aed53f2a54688ee] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:33:50.375 INFO 4486 --- [ocp-schedules-17,10a6559d47afa512,4aed53f2a54688ee] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:33:50.375 INFO 4486 --- [ocp-schedules-17,10a6559d47afa512,4aed53f2a54688ee] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:33:50.376 INFO 4486 --- [ocp-schedules-17,10a6559d47afa512,4aed53f2a54688ee] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:33:50.376 INFO 4486 --- [ocp-schedules-17,10a6559d47afa512,4aed53f2a54688ee] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:33:50.563 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:33:50.563 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:33:50.575 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:33:50.575 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:33:50.575 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:33:50.587 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:33:50.587 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:33:50.587 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:33:50.592 INFO 4486 --- [ocp-schedules-26,a6eac8044a2ba2c3,efc7fe920b1f5e8b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:33:50.871 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9f6ce316ba27c427,66818aac154d9eb7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9f6ce316ba27c427, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:50.871 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9f6ce316ba27c427,66818aac154d9eb7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9f6ce316ba27c427, duration=1 ms] 2024-10-16 13:33:51.045 INFO 4486 --- [ocp-async-4,67476b52b1e00395,c38f4da27bc312e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:51.238 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6da9394ca3472416,7996dcb6baa51288] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6da9394ca3472416, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:51.239 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6da9394ca3472416,7996dcb6baa51288] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6da9394ca3472416, duration=1 ms] 2024-10-16 13:33:51.744 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:52.987 INFO 4486 --- [ocp-async-6,7d0bc4e66dd0bac2,63ee861f4b093d7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:33:52.994 INFO 4486 --- [prometheus-minute-schedule-1,b7c6fe486363bd36,66dd0813e7667cb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:52.999 INFO 4486 --- [prometheus-minute-schedule-1,b7c6fe486363bd36,66dd0813e7667cb7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:53.045 INFO 4486 --- [ocp-async-9,16951d416346c407,63e0b0dfdd9b1cc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:54.418 INFO 4486 --- [ocp-schedules-12,94f2ec584eadf439,bccaff7ace4fe491] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:33:54.418+08:00, tenantSize=5 2024-10-16 13:33:54.422 INFO 4486 --- [ocp-schedules-12,94f2ec584eadf439,bccaff7ace4fe491] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729056834418000 2024-10-16 13:33:54.422 INFO 4486 --- [ocp-schedules-12,94f2ec584eadf439,bccaff7ace4fe491] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729056534347000 2024-10-16 13:33:54.439 INFO 4486 --- [ocp-schedules-12,94f2ec584eadf439,bccaff7ace4fe491] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:33:54.443 INFO 4486 --- [ocp-schedules-12,94f2ec584eadf439,bccaff7ace4fe491] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729056834441000 2024-10-16 13:33:54.443 INFO 4486 --- [ocp-schedules-12,94f2ec584eadf439,bccaff7ace4fe491] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729056534377000 2024-10-16 13:33:54.471 INFO 4486 --- [ocp-schedules-12,94f2ec584eadf439,bccaff7ace4fe491] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:33:54.745 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:55.045 INFO 4486 --- [ocp-async-2,0aef26efc5d3bcc9,f55196a071179088] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:55.994 INFO 4486 --- [prometheus-minute-schedule-1,e89b31a94f1bf4c8,04e6fba5a50e0b33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:56.001 INFO 4486 --- [prometheus-minute-schedule-1,e89b31a94f1bf4c8,04e6fba5a50e0b33] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:57.046 INFO 4486 --- [ocp-async-3,1a62efdcf10cdb05,d976e501293cbb23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:57.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6706bef9d5fe5082,a37ec8ef484601c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6706bef9d5fe5082, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:57.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6706bef9d5fe5082,a37ec8ef484601c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6706bef9d5fe5082, duration=2 ms] 2024-10-16 13:33:57.746 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:33:58.863 INFO 4486 --- [ocp-schedules-26,7d6dcbd5da07953f,59b8e04b7d9d7529] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:33:58.994 INFO 4486 --- [prometheus-minute-schedule-1,00dc6f008749a7d2,d67d078e07cabaf7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056840, interval=60 2024-10-16 13:33:58.999 INFO 4486 --- [prometheus-minute-schedule-1,00dc6f008749a7d2,d67d078e07cabaf7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:33:59.046 INFO 4486 --- [ocp-async-6,6009ff0a614d44cd,3682ba559a1f4d28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:33:59.300 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7bd1e3d90c1f2852,cb08a5e7ad64aca5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7bd1e3d90c1f2852, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:33:59.300 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7bd1e3d90c1f2852,cb08a5e7ad64aca5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7bd1e3d90c1f2852, duration=1 ms] 2024-10-16 13:34:00.014 INFO 4486 --- [ocp-schedules-10,24b75d0b2eb01915,9b51dd7a2cd600c8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:34:00.052 WARN 4486 --- [scanner-13250217-1,24b75d0b2eb01915,a73badf269046117] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:34:00.103 INFO 4486 --- [alarm-detect-1,24b75d0b2eb01915,e8a1d7c1bef2f9fd] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056810 ~ 1729056830 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056825, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056825, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056825=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:34:00.311 INFO 4486 --- [ocp-schedules-20,1576e91f98d6794c,61510679868126d5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:34:00.747 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:00.878 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,21697f1d0f38337c,93f1dd4300bc3f44] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=21697f1d0f38337c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:00.878 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,21697f1d0f38337c,93f1dd4300bc3f44] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=21697f1d0f38337c, duration=1 ms] 2024-10-16 13:34:01.046 INFO 4486 --- [ocp-async-9,7ec77083a7e1b3c3,c3aecfcb48e00a9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:01.114 INFO 4486 --- [ocp-schedules-19,d74627754c61b5e7,21401a190bd28ce1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:34:01.114+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:34:01.118 INFO 4486 --- [ocp-schedules-19,d74627754c61b5e7,21401a190bd28ce1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:34:01.124 INFO 4486 --- [ocp-schedules-19,d74627754c61b5e7,21401a190bd28ce1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:34:01.138 INFO 4486 --- [ocp-schedules-19,d74627754c61b5e7,21401a190bd28ce1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:34:01.154 INFO 4486 --- [ocp-schedules-16,296f6bcb6168df90,dab8de754f6d98ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:34:01.155 INFO 4486 --- [ocp-schedules-16,7db034459abf576c,72b991b4cf48fe5d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:34:01.157 INFO 4486 --- [ocp-schedules-16,7db034459abf576c,72b991b4cf48fe5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:01.191 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.191 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:01.208 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.209 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:01.227 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.228 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.229 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:34:01.231 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.231 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:01.242 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d51e88b1173df154,2cc18a9561cb9381] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d51e88b1173df154, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:01.243 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d51e88b1173df154,2cc18a9561cb9381] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d51e88b1173df154, duration=1 ms] 2024-10-16 13:34:01.262 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.263 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.264 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:01.265 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:34:01.265 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.268 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:34:01.269 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:01.278 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.278 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:01.289 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.298 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.299 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.300 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:01.301 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:34:01.306 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:34:01.312 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:01.314 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:01.337 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.339 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.340 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:34:01.348 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:34:01.382 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:34:01.382 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:34:01.389 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.390 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.391 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:01.395 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.398 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:34:01.402 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:01.429 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.429 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:01.438 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:34:01.447 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.448 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.448 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:34:01.455 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.455 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:01.457 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:34:01.457 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:34:01.479 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.479 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:01.505 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.505 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:01.529 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.529 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:01.556 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:01.581 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.583 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.584 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:01.589 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:34:01.594 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:34:01.598 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:34:01.600 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:34:01.621 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:34:01.770 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:34:01.796 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:01.797 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:01.799 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:34:01.826 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:34:01.826 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:34:01.994 INFO 4486 --- [prometheus-minute-schedule-1,3560e05c8d727717,39c6086d6ff779ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:02.001 INFO 4486 --- [prometheus-minute-schedule-1,3560e05c8d727717,39c6086d6ff779ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:02.066 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:34:02.066 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=7 2024-10-16 13:34:02.066 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1000015 2024-10-16 13:34:02.066 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2024-10-16 13:34:02.066 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2024-10-16 13:34:02.084 WARN 4486 --- [ForkJoinPool.commonPool-worker-30,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:34:02.090 WARN 4486 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:34:02.101 WARN 4486 --- [ForkJoinPool.commonPool-worker-19,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.134 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 7 agent sql collect property 2024-10-16 13:34:02.135 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:02.139 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:02.148 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:34:02.148 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:34:02.149 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:34:02.149 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2024-10-16 13:34:02.160 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2024-10-16 13:34:02.163 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2024-10-16 13:34:02.164 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:02.167 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1000015 agent sql collect property 2024-10-16 13:34:02.168 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:02.173 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2024-10-16 13:34:02.176 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:02.178 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:02.179 INFO 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 7, do nothing 2024-10-16 13:34:02.179 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:02.181 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:02.199 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313637), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.184), (ob.run.path=/root/oceanbase), (ob.install.path=/root/oceanbase), (ob.data.path=/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=OceanBase), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local ser 2024-10-16 13:34:02.212 ERROR 4486 --- [ocp-schedules-20,e923e32973e9a0a4,565435f3155ac0cd] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor1649.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:Unexpected error: key monagent.ob.cgroup.status is not found in config properties! at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:476) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:286) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:114) at com.oceanbase.ocp.executor.executor.AgentExecutor.updateAgentConfig(AgentExecutor.java:193) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentProperties(HostAgentConfigManagerImpl.java:188) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentConfig(HostAgentConfigManagerImpl.java:196) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$FastClassBySpringCGLIB$$798eb4cd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$EnhancerBySpringCGLIB$$eb50b66a.refreshAgentConfig() at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.refreshHostAgentConfig(HostAgentServiceImpl.java:734) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$5f28aa5.refreshHostAgentConfig() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$refreshAgentConfig$0(HostSchedules.java:92) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:34:02.212 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313639), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.178), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data), (monagent.cluster.id=5), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=obce_3zones), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=2), (monagent.obproxy.cluster.name=obce_3zones), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=10.197.49.178), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLev 2024-10-16 13:34:02.216 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313644), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=172.28.46.152), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1000003), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=glob_3zone), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=1000002), (monagent.obproxy.cluster.name=xc_OB43), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=172.28.46.152), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition fro 2024-10-16 13:34:02.221 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2024-10-16 13:34:02.224 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1000015, do nothing 2024-10-16 13:34:02.988 INFO 4486 --- [ocp-async-7,b391cdedd7795886,0b620d282f0ea7b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:34:03.046 INFO 4486 --- [ocp-async-4,ace56f209a7ddfc0,8fa22abf7205a644] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:03.748 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:04.994 INFO 4486 --- [prometheus-minute-schedule-1,f74d1c3a17000476,e9f3b6bfa9918dc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:05.001 INFO 4486 --- [prometheus-minute-schedule-1,f74d1c3a17000476,e9f3b6bfa9918dc8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:05.049 INFO 4486 --- [ocp-async-8,5b2208273749f7db,788e8e593bf2d3cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:06.749 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:07.050 INFO 4486 --- [ocp-async-9,0b1d0d56dc455b9a,428117d758abaf31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:07.066 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:34:07.067 INFO 4486 --- [ocp-schedules-1,962a5d308f37f013,e8dee96e58e3b5c3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:34:07.068 INFO 4486 --- [ocp-schedules-1,962a5d308f37f013,e8dee96e58e3b5c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:07.075 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b591cbcbadce3513,6661c991976a3de6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b591cbcbadce3513, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:07.075 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b591cbcbadce3513,6661c991976a3de6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b591cbcbadce3513, duration=1 ms] 2024-10-16 13:34:07.081 INFO 4486 --- [ocp-schedules-3,5c5b3a3b9cbc6afa,2f9a0cd81b84c7e8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:34:07.083 INFO 4486 --- [ocp-schedules-3,5c5b3a3b9cbc6afa,2f9a0cd81b84c7e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:07.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:34:07.089 INFO 4486 --- [ocp-schedules-23,b94a83284608d332,8f05b4c57614b465] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:34:07.089 INFO 4486 --- [ocp-schedules-26,15b2f73af96e50a4,a9d162d79c56f1be] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:34:07.089 INFO 4486 --- [ocp-schedules-14,e695dc448db5c01a,b64b99d9983cad92] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:34:07.090 INFO 4486 --- [host-check-clockdiff1,b94a83284608d332,2fb618084ff2b230] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:34:07.090 INFO 4486 --- [host-check-agent-status1,e695dc448db5c01a,aa8eabfb3b2dae31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:34:07.090 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:34:07.090 INFO 4486 --- [host-check-agent-status2,e695dc448db5c01a,d4d5964ad7872407] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:34:07.091 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:34:07.091 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:34:07.091 INFO 4486 --- [host-check-agent-status3,e695dc448db5c01a,ab932ad5ef284bdc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:34:07.091 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:34:07.092 INFO 4486 --- [host-check-agent-status4,e695dc448db5c01a,ea6c0c6796cdb016] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:34:07.092 INFO 4486 --- [host-check-clockdiff2,b94a83284608d332,36a6914d0d84208a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:34:07.093 INFO 4486 --- [host-check-clockdiff3,b94a83284608d332,fa33640e4289e8e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:34:07.094 INFO 4486 --- [host-check-clockdiff4,b94a83284608d332,fbc701b06003540d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:34:07.096 INFO 4486 --- [host-check-agent-status1,e695dc448db5c01a,aa8eabfb3b2dae31] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:07.097 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:07.097 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:07.098 INFO 4486 --- [host-check-agent-status2,e695dc448db5c01a,d4d5964ad7872407] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:07.099 INFO 4486 --- [host-check-agent-status1,e695dc448db5c01a,aa8eabfb3b2dae31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:07.099 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:07.099 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:07.099 INFO 4486 --- [host-check-agent-status2,e695dc448db5c01a,d4d5964ad7872407] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:07.099 INFO 4486 --- [host-check-agent-status3,e695dc448db5c01a,ab932ad5ef284bdc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:07.099 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:07.100 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:07.100 INFO 4486 --- [host-check-agent-status3,e695dc448db5c01a,ab932ad5ef284bdc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:07.101 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:07.101 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:07.102 INFO 4486 --- [host-check-agent-status4,e695dc448db5c01a,ea6c0c6796cdb016] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:07.117 INFO 4486 --- [host-check-agent-status4,e695dc448db5c01a,ea6c0c6796cdb016] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:07.117 INFO 4486 --- [host-check-clockdiff3,b94a83284608d332,fa33640e4289e8e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056847 3 3 , error= 2024-10-16 13:34:07.117 INFO 4486 --- [host-check-clockdiff4,b94a83284608d332,fbc701b06003540d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056847 3 3 , error= 2024-10-16 13:34:07.120 INFO 4486 --- [host-check-clockdiff2,b94a83284608d332,36a6914d0d84208a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056847 3 3 , error= 2024-10-16 13:34:07.123 INFO 4486 --- [host-check-clockdiff1,b94a83284608d332,2fb618084ff2b230] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056847 0 0 , error= 2024-10-16 13:34:07.125 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.125 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:07.126 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:34:07.142 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.142 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:34:07.142 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:34:07.143 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:34:07.143 INFO 4486 --- [host-check-agent-status2,e695dc448db5c01a,d4d5964ad7872407] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:34:07.143 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:07.143 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:34:07.145 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:34:07.144664943+08:00 2024-10-16 13:34:07.145 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:34:07.148032615+08:00 2024-10-16 13:34:07.145 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:34:07.147732856+08:00 2024-10-16 13:34:07.145 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:34:07.145+08:00 2024-10-16 13:34:07.145 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:34:07.145+08:00 2024-10-16 13:34:07.146 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:34:07.148777897+08:00 2024-10-16 13:34:07.146 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:34:07.146+08:00 2024-10-16 13:34:07.146 INFO 4486 --- [host-check-agent-status4,e695dc448db5c01a,ea6c0c6796cdb016] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:34:07.146 INFO 4486 --- [host-check-agent-status1,e695dc448db5c01a,aa8eabfb3b2dae31] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:34:07.146 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:34:07.145+08:00 2024-10-16 13:34:07.146 INFO 4486 --- [host-check-agent-status3,e695dc448db5c01a,ab932ad5ef284bdc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:34:07.147 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:34:07.148 INFO 4486 --- [host-check-agent-status2,e695dc448db5c01a,d4d5964ad7872407] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:34:07.148 INFO 4486 --- [host-check-agent-status2,e695dc448db5c01a,d4d5964ad7872407] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:34:07.148 INFO 4486 --- [host-check-agent-status2,e695dc448db5c01a,d4d5964ad7872407] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:34:07.148 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:34:07.145+08:00 and status AVAILABLE 2024-10-16 13:34:07.148 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:34:07.145+08:00 and status AVAILABLE 2024-10-16 13:34:07.148 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:34:07.146+08:00 and status AVAILABLE 2024-10-16 13:34:07.149 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:34:07.145+08:00 and status AVAILABLE 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status4,e695dc448db5c01a,ea6c0c6796cdb016] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status1,e695dc448db5c01a,aa8eabfb3b2dae31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status4,e695dc448db5c01a,ea6c0c6796cdb016] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status1,e695dc448db5c01a,aa8eabfb3b2dae31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status4,e695dc448db5c01a,ea6c0c6796cdb016] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status1,e695dc448db5c01a,aa8eabfb3b2dae31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status3,e695dc448db5c01a,ab932ad5ef284bdc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status3,e695dc448db5c01a,ab932ad5ef284bdc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:34:07.150 INFO 4486 --- [host-check-agent-status3,e695dc448db5c01a,ab932ad5ef284bdc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:34:07.151 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:34:07.151 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:34:07.151 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:34:07.151 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:34:07.158 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:07.158 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.159 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:07.159 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:07.160 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:07.160 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:07.160 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:07.161 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:07.161 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:07.163 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:34:07.170 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.171 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:07.179 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:07.180 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.181 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:07.181 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:07.181 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:07.185 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:07.190 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.191 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:07.200 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.201 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:34:07.208 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:34:07.213 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:07.213 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:34:07.214 INFO 4486 --- [ocp-schedules-4,3bf793c6b53a676a,eec51dd28615cd01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:34:07.217 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:34:07.220 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:34:07.226 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:34:07.232 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:07.232 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:34:07.245 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:07.245 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:34:07.250 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:07.258 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:34:07.260 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:34:07.271 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:34:07.274 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:07.280 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:34:07.288 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:07.289 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:34:07.297 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:34:07.304 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:07.307 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:34:07.315 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:34:07.322 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:07.325 INFO 4486 --- [host-check-host-status3,15b2f73af96e50a4,d0bf9436255d94a4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:34:07.327 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:34:07.338 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:34:07.342 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:34:07.352 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:07.352 INFO 4486 --- [host-check-host-status4,15b2f73af96e50a4,1e6ecb76c38ca8d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:34:07.363 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:34:07.375 INFO 4486 --- [host-check-host-status2,15b2f73af96e50a4,2cd5d9d0f31b66a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:34:07.405 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:34:07.419 INFO 4486 --- [host-check-host-status1,15b2f73af96e50a4,0559c4a55ea59302] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:34:07.994 INFO 4486 --- [prometheus-minute-schedule-1,7950cce894a7ee75,ac9fd4765fe9a5a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:08.002 INFO 4486 --- [prometheus-minute-schedule-1,7950cce894a7ee75,ac9fd4765fe9a5a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:09.050 INFO 4486 --- [ocp-async-7,2493b6a2ba6c5514,591763b5f77cde3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:09.188 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,fb4499a4791288b1,1bdce9d04ae6280b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=fb4499a4791288b1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:34:09.189 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,fb4499a4791288b1,1bdce9d04ae6280b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=fb4499a4791288b1, duration=3 ms] 2024-10-16 13:34:09.305 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9b2336cb79ad896d,f2e59314070232b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9b2336cb79ad896d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:09.305 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9b2336cb79ad896d,f2e59314070232b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9b2336cb79ad896d, duration=1 ms] 2024-10-16 13:34:09.689 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1bd75965a23764ef,e039ced8ee7a8fd8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=1bd75965a23764ef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:34:09.689 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1bd75965a23764ef,e039ced8ee7a8fd8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=1bd75965a23764ef, duration=1 ms] 2024-10-16 13:34:09.749 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:10.006 INFO 4486 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:34:10.007 INFO 4486 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:34:10.027 INFO 4486 --- [Check obproxy1,0ed71d26a994c7ff,2c3e212f4ebca449] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:34:10.028 INFO 4486 --- [Check obproxy1,0ed71d26a994c7ff,2c3e212f4ebca449] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:10.028 INFO 4486 --- [Check obproxy2,aec02c5f8b35f0f3,e67dcf7d87e7f066] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:34:10.028 INFO 4486 --- [Check obproxy2,aec02c5f8b35f0f3,e67dcf7d87e7f066] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:10.032 INFO 4486 --- [Collect obproxy2,aa3bbb1908d20657,e443e22ebf7f89f1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:34:10.032 INFO 4486 --- [Collect obproxy1,5263156c99545d94,ed038d7cd1f4dac4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:34:10.032 INFO 4486 --- [Collect obproxy2,aa3bbb1908d20657,e443e22ebf7f89f1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:10.032 INFO 4486 --- [Collect obproxy1,5263156c99545d94,ed038d7cd1f4dac4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:10.052 INFO 4486 --- [Check obproxy2,aec02c5f8b35f0f3,e67dcf7d87e7f066] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:34:10.052 INFO 4486 --- [Check obproxy2,aec02c5f8b35f0f3,e67dcf7d87e7f066] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:34:10.053 INFO 4486 --- [Collect obproxy2,aa3bbb1908d20657,e443e22ebf7f89f1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:34:10.056 INFO 4486 --- [Check obproxy1,0ed71d26a994c7ff,2c3e212f4ebca449] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:34:10.056 INFO 4486 --- [Check obproxy1,0ed71d26a994c7ff,2c3e212f4ebca449] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:34:10.058 INFO 4486 --- [Collect obproxy2,aa3bbb1908d20657,e443e22ebf7f89f1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:34:10.061 INFO 4486 --- [Collect obproxy1,5263156c99545d94,ed038d7cd1f4dac4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:34:10.063 INFO 4486 --- [Collect obproxy2,aa3bbb1908d20657,e443e22ebf7f89f1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:34:10.066 INFO 4486 --- [Collect obproxy1,5263156c99545d94,ed038d7cd1f4dac4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:34:10.070 INFO 4486 --- [Collect obproxy1,5263156c99545d94,ed038d7cd1f4dac4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:34:10.183 INFO 4486 --- [ocp-schedules-29,7b2813df23fc7f94,9af0d9c9dc416b16] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:34:10.194 WARN 4486 --- [scanner-13250419-1,7b2813df23fc7f94,5b4d33c25fa5ac3d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:34:10.287 INFO 4486 --- [alarm-detect-3,7b2813df23fc7f94,767dd0e77c270d11] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056825 ~ 1729056840 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056840, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056840, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056840=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:34:10.317 INFO 4486 --- [ocp-schedules-5,6ff7041444b9713f,a5efde0d36a548c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:34:10.882 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,53837c7ce203de4d,4b1f41600c747818] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=53837c7ce203de4d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:10.882 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,53837c7ce203de4d,4b1f41600c747818] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=53837c7ce203de4d, duration=1 ms] 2024-10-16 13:34:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:34:10.994 INFO 4486 --- [prometheus-minute-schedule-1,6575bd68d08a0fea,08d0bc84bc78c949] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:11.003 INFO 4486 --- [prometheus-minute-schedule-1,6575bd68d08a0fea,08d0bc84bc78c949] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:11.014 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:34:11.050 INFO 4486 --- [ocp-async-3,e3a7966c7678a5c0,6cf8356d1b6fa742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:11.246 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dd12dab510a770fa,c237f7e03bba313e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dd12dab510a770fa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:11.246 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dd12dab510a770fa,c237f7e03bba313e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dd12dab510a770fa, duration=1 ms] 2024-10-16 13:34:12.750 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:12.988 INFO 4486 --- [ocp-async-8,9c4b78bc56e51940,49237ac633ef7d69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:34:13.050 INFO 4486 --- [ocp-async-5,bb6a8eb0da1884eb,5889523d092c5d79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:13.199 ERROR 4486 --- [metric-archive-second-30,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '586679390194700288-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.198340] [YB420AC531B8-00060B7DB03B22F4-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '586679390194700288-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.198340] [YB420AC531B8-00060B7DB03B22F4-0-0] 2024-10-16 13:34:13.204 ERROR 4486 --- [metric-archive-second-16,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2323956) Duplicate entry '669741696482349056-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.203724] [YB420AC531B8-00060B7DBC38716E-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '669741696482349056-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.203724] [YB420AC531B8-00060B7DBC38716E-0-0] 2024-10-16 13:34:13.207 ERROR 4486 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=4117402) Duplicate entry '587073990835113984-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.206625] [YB420AC531B8-00060B7DB4195958-0-0]; nested exception is java.sql.BatchUpdateException: (conn=4117402) Duplicate entry '587073990835113984-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.206625] [YB420AC531B8-00060B7DB4195958-0-0] 2024-10-16 13:34:13.208 ERROR 4486 --- [metric-archive-second-9,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=70525) Duplicate entry '587073990944165888-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.207117] [YB420AC531B8-00060B7D84EAACC1-0-0]; nested exception is java.sql.BatchUpdateException: (conn=70525) Duplicate entry '587073990944165888-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.207117] [YB420AC531B8-00060B7D84EAACC1-0-0] 2024-10-16 13:34:13.211 ERROR 4486 --- [metric-archive-second-14,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990814) Duplicate entry '586774769288417280-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.211206] [YB420AC531B8-00060B7DB46FB53C-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990814) Duplicate entry '586774769288417280-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.211206] [YB420AC531B8-00060B7DB46FB53C-0-0] 2024-10-16 13:34:13.212 ERROR 4486 --- [metric-archive-second-13,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620709) Duplicate entry '586679389997568000-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.211955] [YB420AC531B8-00060B7DB76883FC-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620709) Duplicate entry '586679389997568000-1729056780' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:34:13.211955] [YB420AC531B8-00060B7DB76883FC-0-0] 2024-10-16 13:34:13.387 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056780, nodeCount=286, affectRows=286 2024-10-16 13:34:13.389 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056780, nodeCount=286, validCount=286 2024-10-16 13:34:13.566 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056780, nodeCount=623, affectRows=623 2024-10-16 13:34:13.567 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056780, nodeCount=599, affectRows=599 2024-10-16 13:34:13.569 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056780, nodeCount=623, validCount=623 2024-10-16 13:34:13.569 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056780, nodeCount=598, validCount=598 2024-10-16 13:34:13.582 INFO 4486 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056780, nodeCount=613, affectRows=613 2024-10-16 13:34:13.586 INFO 4486 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056780, nodeCount=612, validCount=612 2024-10-16 13:34:13.601 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056780, nodeCount=679, affectRows=679 2024-10-16 13:34:13.602 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056780, nodeCount=650, affectRows=650 2024-10-16 13:34:13.603 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056780, nodeCount=678, validCount=678 2024-10-16 13:34:13.605 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056780, nodeCount=650, validCount=650 2024-10-16 13:34:13.994 INFO 4486 --- [prometheus-minute-schedule-1,b2103faaef64c429,76252ebdbec24056] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:14.002 INFO 4486 --- [prometheus-minute-schedule-1,b2103faaef64c429,76252ebdbec24056] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:14.417 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056780, nodeCount=477, affectRows=477 2024-10-16 13:34:15.051 INFO 4486 --- [ocp-async-1,c4c87b69d4dc3fc2,868ea0fdee1fc05e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:15.107 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056780, nodeCount=477, validCount=477 2024-10-16 13:34:15.751 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:16.994 INFO 4486 --- [prometheus-minute-schedule-1,8ee707670889f46b,6c45725e41284438] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:17.000 INFO 4486 --- [prometheus-minute-schedule-1,8ee707670889f46b,6c45725e41284438] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:17.051 INFO 4486 --- [ocp-async-6,92d50799d2977eee,8fce284b6e61288f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:17.069 INFO 4486 --- [ocp-schedules-1,962a5d308f37f013,e8dee96e58e3b5c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:34:17.079 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,528220d35d454a1e,91944af269b578be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=528220d35d454a1e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:17.079 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,528220d35d454a1e,91944af269b578be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=528220d35d454a1e, duration=1 ms] 2024-10-16 13:34:17.092 INFO 4486 --- [ocp-schedules-25,37ce06d0939c418d,991496e79670ff2f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:34:17.148 INFO 4486 --- [ocp-schedules-3,5c5b3a3b9cbc6afa,2f9a0cd81b84c7e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:34:18.752 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:18.867 INFO 4486 --- [ocp-schedules-12,40726ad3fc099b19,dad39e543f0cde4b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:34:19.051 INFO 4486 --- [ocp-async-10,a94a87494b0ba66f,a47390920a7192b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:19.309 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,353afdbb95a943cb,e06a3a646a25e69f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=353afdbb95a943cb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:19.309 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,353afdbb95a943cb,e06a3a646a25e69f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=353afdbb95a943cb, duration=1 ms] 2024-10-16 13:34:19.994 INFO 4486 --- [prometheus-minute-schedule-1,9b73ac0fbb86ec17,02bf4ea451df4c63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:20.000 INFO 4486 --- [prometheus-minute-schedule-1,9b73ac0fbb86ec17,02bf4ea451df4c63] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:20.326 INFO 4486 --- [ocp-schedules-1,1210b31fd451de32,6a8e617ca2a49a5a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:34:20.394 INFO 4486 --- [ocp-schedules-12,f5c0b076c3a05abd,8c02cf9ff92ffb69] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:34:20.437 WARN 4486 --- [scanner-13251615-1,f5c0b076c3a05abd,c1bd9033f6ce0630] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:34:20.810 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:34:20.887 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,684a169d5c3af374,5330829e3c70baac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=684a169d5c3af374, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:20.888 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,684a169d5c3af374,5330829e3c70baac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=684a169d5c3af374, duration=1 ms] 2024-10-16 13:34:20.996 INFO 4486 --- [prometheus-cache-monitor1,5e35fbcb30d8f4a4,70c17d2db63e39e3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:34:20.996 INFO 4486 --- [prometheus-cache-monitor1,5e35fbcb30d8f4a4,70c17d2db63e39e3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:34:21.052 INFO 4486 --- [ocp-async-5,d309c6d8119d2e5f,684ea217f7348e1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:21.250 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ed39a160ce8f30a2,8b772eff3470c976] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ed39a160ce8f30a2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:21.250 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ed39a160ce8f30a2,8b772eff3470c976] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ed39a160ce8f30a2, duration=1 ms] 2024-10-16 13:34:21.752 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:22.764 INFO 4486 --- [ocp-schedules-24,5be586225b5b6432,5d8293a38a5a8d07] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:34:22.988 INFO 4486 --- [ocp-async-7,d98b5aeae0f5d76f,5315e14a72d4f3da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:34:22.994 INFO 4486 --- [prometheus-minute-schedule-1,77cc7f3bdfcf0c68,aace1984d36ce09b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:22.999 INFO 4486 --- [prometheus-minute-schedule-1,77cc7f3bdfcf0c68,aace1984d36ce09b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:23.052 INFO 4486 --- [ocp-async-4,19c3070b8c44a526,ceb1b4c01d9906ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:23.456 INFO 4486 --- [ocp-schedules-4,2ddc4d07900b628e,8d720026b17f771d] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:34:23.458 INFO 4486 --- [ocp-schedules-4,2ddc4d07900b628e,8d720026b17f771d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:23.458 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:34:23.459 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:34:23.459 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:34:24.083 INFO 4486 --- [ocp-schedules-6,29ecf3373092a54d,aa2a9917ae7c1f4c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:34:24.085 INFO 4486 --- [ocp-schedules-6,29ecf3373092a54d,aa2a9917ae7c1f4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:24.116 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:34:24.117 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.121 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:34:24.130 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.130 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:34:24.139 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.139 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:34:24.139 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:34:24.140 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.144 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:34:24.147 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.147 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:34:24.155 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.155 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:34:24.159 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.160 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:34:24.164 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.164 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:24.174 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.175 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.175 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:34:24.176 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:24.176 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:34:24.176 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.181 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:34:24.184 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.185 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:34:24.186 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.187 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:24.192 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.193 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:34:24.194 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.194 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:34:24.197 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.200 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:24.204 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.205 INFO 4486 --- [ocp-schedules-23,33507a7571f2fb84,f299489dc41982bf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:34:24.205 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:34:24.209 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.210 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:34:24.211 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.211 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.212 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.213 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.214 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.215 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.216 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.217 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.218 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.218 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.218 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:34:24.219 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.220 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.220 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.221 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:34:24.221 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.222 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.222 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.223 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:34:24.223 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.224 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.225 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.226 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.227 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.227 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.228 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.229 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.230 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.231 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.231 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.232 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:34:24.232 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.233 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:34:24.234 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:34:24.242 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.244 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:34:24.244 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.245 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.246 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:24.248 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.251 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:24.252 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.252 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.253 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:34:24.254 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.256 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.257 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.257 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:24.258 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.262 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.263 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.266 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:34:24.267 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.269 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.270 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:34:24.270 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:34:24.271 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.272 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.273 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.274 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:24.274 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.276 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.279 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.280 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.282 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.283 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.286 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:34:24.289 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:34:24.306 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.307 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.310 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.311 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.313 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.314 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.316 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:24.317 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.318 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.318 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.319 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:34:24.321 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.322 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.324 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.325 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.328 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.329 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:34:24.332 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.333 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:34:24.334 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.334 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:34:24.334 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.334 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:34:24.334 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.334 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:34:24.336 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.337 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:34:24.337 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:34:24.337 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:34:24.339 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:34:24.342 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:34:24.400 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:24.401 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.403 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:34:24.412 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.412 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:34:24.412 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.412 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:34:24.412 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:34:24.412 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:34:24.416 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:34:24.416 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:34:24.524 INFO 4486 --- [ocp-schedules-14,9ab0c30a1521b8fa,c2ae8b76c8616a0d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:34:24.524 INFO 4486 --- [ocp-schedules-14,9ab0c30a1521b8fa,c2ae8b76c8616a0d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:34:24.557 INFO 4486 --- [Collect all tenants major compactions1,9ab0c30a1521b8fa,a387314fd6fe3c31] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:34:24.557 INFO 4486 --- [Collect all tenants major compactions2,9ab0c30a1521b8fa,e8877145076c3ce0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:34:24.557 INFO 4486 --- [Collect all tenants major compactions3,9ab0c30a1521b8fa,381a905787af6682] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:34:24.558 INFO 4486 --- [Collect all tenants major compactions5,9ab0c30a1521b8fa,1785ca36075f1461] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:34:24.558 INFO 4486 --- [Collect all tenants major compactions4,9ab0c30a1521b8fa,6ec36c048063ca6f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:34:24.572 INFO 4486 --- [Collect all tenants major compactions2,9ab0c30a1521b8fa,e8877145076c3ce0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.572 INFO 4486 --- [Collect all tenants major compactions1,9ab0c30a1521b8fa,a387314fd6fe3c31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.574 INFO 4486 --- [Collect all tenants major compactions1,9ab0c30a1521b8fa,a387314fd6fe3c31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:34:24.574 INFO 4486 --- [Collect all tenants major compactions2,9ab0c30a1521b8fa,e8877145076c3ce0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:34:24.577 INFO 4486 --- [Collect all tenants major compactions3,9ab0c30a1521b8fa,381a905787af6682] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.578 INFO 4486 --- [Collect all tenants major compactions1,9ab0c30a1521b8fa,a387314fd6fe3c31] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:34:24.578 INFO 4486 --- [Collect all tenants major compactions2,9ab0c30a1521b8fa,e8877145076c3ce0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:34:24.578 INFO 4486 --- [Collect all tenants major compactions2,9ab0c30a1521b8fa,e8877145076c3ce0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:34:24.578 INFO 4486 --- [Collect all tenants major compactions1,9ab0c30a1521b8fa,a387314fd6fe3c31] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:34:24.579 INFO 4486 --- [Collect all tenants major compactions3,9ab0c30a1521b8fa,381a905787af6682] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:34:24.584 INFO 4486 --- [Collect all tenants major compactions3,9ab0c30a1521b8fa,381a905787af6682] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:34:24.584 INFO 4486 --- [Collect all tenants major compactions3,9ab0c30a1521b8fa,381a905787af6682] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:34:24.589 INFO 4486 --- [Collect all tenants major compactions5,9ab0c30a1521b8fa,1785ca36075f1461] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.590 INFO 4486 --- [Collect all tenants major compactions4,9ab0c30a1521b8fa,6ec36c048063ca6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:24.591 INFO 4486 --- [Collect all tenants major compactions5,9ab0c30a1521b8fa,1785ca36075f1461] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:34:24.591 INFO 4486 --- [Collect all tenants major compactions4,9ab0c30a1521b8fa,6ec36c048063ca6f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:34:24.595 INFO 4486 --- [Collect all tenants major compactions4,9ab0c30a1521b8fa,6ec36c048063ca6f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:34:24.595 INFO 4486 --- [Collect all tenants major compactions4,9ab0c30a1521b8fa,6ec36c048063ca6f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:34:24.595 INFO 4486 --- [Collect all tenants major compactions5,9ab0c30a1521b8fa,1785ca36075f1461] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:34:24.595 INFO 4486 --- [Collect all tenants major compactions5,9ab0c30a1521b8fa,1785ca36075f1461] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:34:24.597 INFO 4486 --- [ocp-schedules-14,9ab0c30a1521b8fa,c2ae8b76c8616a0d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:34:24.753 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:25.052 INFO 4486 --- [ocp-async-8,a26ab060cd64271a,ee076a670500c99f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:25.994 INFO 4486 --- [prometheus-minute-schedule-1,47ca3f9a2be0fa70,a68758e0a2e27ada] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:26.002 INFO 4486 --- [prometheus-minute-schedule-1,47ca3f9a2be0fa70,a68758e0a2e27ada] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:26.896 INFO 4486 --- [ocp-schedules-1,1d63942e64795bbc,db0183a942b87cfb] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:34:26.896+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:34:26.898 INFO 4486 --- [ocp-schedules-1,1d63942e64795bbc,db0183a942b87cfb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:34:26.899 INFO 4486 --- [ocp-schedules-1,1d63942e64795bbc,db0183a942b87cfb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:34:26.899 INFO 4486 --- [ocp-schedules-1,1d63942e64795bbc,db0183a942b87cfb] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:34:27.052 INFO 4486 --- [ocp-async-2,68deab0a634735ac,a1d9fb7a2246cce4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:27.086 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,738b8a729e8a53f9,c2625bfbe08871e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=738b8a729e8a53f9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:27.086 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,738b8a729e8a53f9,c2625bfbe08871e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=738b8a729e8a53f9, duration=2 ms] 2024-10-16 13:34:27.754 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:28.994 INFO 4486 --- [prometheus-minute-schedule-1,d7e087efe69878c0,c44bc622a52101d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:29.000 INFO 4486 --- [prometheus-minute-schedule-1,d7e087efe69878c0,c44bc622a52101d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:29.052 INFO 4486 --- [ocp-async-3,257525234dd4b6a7,c938ff37e6f016c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:29.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d7d91ac316b6ce9c,b07918239d60f585] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d7d91ac316b6ce9c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:29.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d7d91ac316b6ce9c,b07918239d60f585] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d7d91ac316b6ce9c, duration=1 ms] 2024-10-16 13:34:29.332 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3127de44fb42ed21,f060f001483b27a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=3127de44fb42ed21, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:34:29.333 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3127de44fb42ed21,f060f001483b27a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=3127de44fb42ed21, duration=2 ms] 2024-10-16 13:34:30.103 INFO 4486 --- [prometheus-second-schedule-1,afd8677feec54125,51e5e3dc29d46535] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:34:30.330 INFO 4486 --- [ocp-schedules-23,d6ed23690f917fd6,55e84358a81c7c7a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:34:30.648 INFO 4486 --- [ocp-schedules-18,01f1f2cc6a365bd1,0fe95ccf5087edc0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:34:30.685 WARN 4486 --- [scanner-13251843-1,01f1f2cc6a365bd1,1934f839c8962e5a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:34:30.756 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:30.776 INFO 4486 --- [alarm-detect-13,01f1f2cc6a365bd1,baae8a673e41e27f] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056840 ~ 1729056860 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056855, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056855, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056855=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:34:30.892 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,97f29420fd72fc70,e6d0dbb2f8400189] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=97f29420fd72fc70, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:30.892 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,97f29420fd72fc70,e6d0dbb2f8400189] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=97f29420fd72fc70, duration=1 ms] 2024-10-16 13:34:31.052 INFO 4486 --- [ocp-async-6,0d18d8d35bbc5429,d169a3805972a2e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:31.255 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,29042a77db89c5f4,2ec5d4d017a88b73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=29042a77db89c5f4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:31.255 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,29042a77db89c5f4,2ec5d4d017a88b73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=29042a77db89c5f4, duration=1 ms] 2024-10-16 13:34:31.826 INFO 4486 --- [ocp-schedules-16,7db034459abf576c,72b991b4cf48fe5d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:34:31.827 INFO 4486 --- [ocp-schedules-16,60111c7cfacea21e,ddff8b96e94d7c82] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:34:31.829 INFO 4486 --- [ocp-schedules-16,60111c7cfacea21e,ddff8b96e94d7c82] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:31.861 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:31.862 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:31.874 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:31.875 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:31.897 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:31.899 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:31.899 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:31.900 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:31.900 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:34:31.938 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:31.939 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:31.939 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:31.940 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:34:31.941 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:31.944 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:34:31.946 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:31.957 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:31.957 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:31.969 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:31.980 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:31.980 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:31.981 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:31.982 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:34:31.989 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:34:31.993 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:31.994 INFO 4486 --- [prometheus-minute-schedule-1,39f2f0fa521caaf1,768c55785acaa6f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:31.995 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:31.999 INFO 4486 --- [prometheus-minute-schedule-1,39f2f0fa521caaf1,768c55785acaa6f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:32.020 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:32.022 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:32.023 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:34:32.029 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:34:32.072 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:34:32.072 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:34:32.078 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:32.079 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:32.080 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:34:32.084 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:32.087 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:34:32.090 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:32.111 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:34:32.122 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:32.123 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:32.123 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:32.123 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:32.124 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:34:32.134 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:34:32.134 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:34:32.154 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:32.154 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:32.187 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:32.187 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:32.215 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:32.216 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:34:32.242 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:32.242 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:34:32.268 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:34:32.293 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:32.295 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:32.296 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:32.300 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:34:32.305 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:34:32.309 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:34:32.311 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:34:32.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:34:32.474 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:34:32.506 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:32.508 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:32.509 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:34:32.539 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:34:32.539 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:34:32.989 INFO 4486 --- [ocp-async-8,be46ccfb41886687,85a255647a1060ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:34:33.053 INFO 4486 --- [ocp-async-5,48ad7ddaeff2178e,fa3fc8c6a2038e35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:33.459 INFO 4486 --- [ocp-schedules-4,2ddc4d07900b628e,8d720026b17f771d] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:34:33.757 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:34.994 INFO 4486 --- [prometheus-minute-schedule-1,a263fc7dd06f32d9,6c63610960fa8929] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:35.006 INFO 4486 --- [prometheus-minute-schedule-1,a263fc7dd06f32d9,6c63610960fa8929] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:35.053 INFO 4486 --- [ocp-async-7,3c61665541b3c562,2c38fdbbffeb011f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:36.758 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:37.053 INFO 4486 --- [ocp-async-4,25967c5dc211b40a,0d0b205f372eb276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:37.063 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:34:37.064 INFO 4486 --- [ocp-schedules-11,683172986ef79b69,099c5f0df1b0fc54] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:34:37.065 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:34:37.066 INFO 4486 --- [ocp-schedules-24,b186d762d344c687,8c153f925397b103] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:34:37.068 INFO 4486 --- [ocp-schedules-24,b186d762d344c687,8c153f925397b103] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:37.068 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:34:37.071 INFO 4486 --- [ocp-schedules-11,683172986ef79b69,099c5f0df1b0fc54] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:34:37.081 INFO 4486 --- [ocp-schedules-4,633e7c636a764ed2,e48d5904caec30ac] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:34:37.084 INFO 4486 --- [ocp-schedules-4,633e7c636a764ed2,e48d5904caec30ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:34:37.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:34:37.092 INFO 4486 --- [ocp-schedules-17,415526baea8672c8,ae16ad0b4009e1b7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:34:37.092 INFO 4486 --- [ocp-schedules-26,72270bb42449cbc6,0396a3712d0c0030] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:34:37.093 INFO 4486 --- [host-check-agent-status1,415526baea8672c8,064580c53141578d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:34:37.093 INFO 4486 --- [host-check-agent-status2,415526baea8672c8,e01522ae085417bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:34:37.093 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:34:37.093 INFO 4486 --- [host-check-agent-status3,415526baea8672c8,ae96d7f4641d3fe1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:34:37.094 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:34:37.095 INFO 4486 --- [ocp-schedules-1,2b6228a6cbbe1e59,6dde3bbf2a3f2a2b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:34:37.095 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:34:37.095 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:34:37.095 INFO 4486 --- [host-check-agent-status4,415526baea8672c8,cce73e791c99514a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:34:37.098 INFO 4486 --- [host-check-clockdiff3,2b6228a6cbbe1e59,bf8e3a58076443ba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:34:37.098 INFO 4486 --- [host-check-clockdiff2,2b6228a6cbbe1e59,3817c42fb2edb2d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:34:37.098 INFO 4486 --- [host-check-clockdiff1,2b6228a6cbbe1e59,80a753973361fd6d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:34:37.099 INFO 4486 --- [host-check-clockdiff4,2b6228a6cbbe1e59,f90cad47c1dce402] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:34:37.100 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:37.101 INFO 4486 --- [host-check-agent-status2,415526baea8672c8,e01522ae085417bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:37.101 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:37.102 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:37.102 INFO 4486 --- [host-check-agent-status1,415526baea8672c8,064580c53141578d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:37.103 INFO 4486 --- [host-check-agent-status2,415526baea8672c8,e01522ae085417bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:37.104 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:37.104 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:37.104 INFO 4486 --- [host-check-agent-status4,415526baea8672c8,cce73e791c99514a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:37.104 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:37.104 INFO 4486 --- [host-check-agent-status3,415526baea8672c8,ae96d7f4641d3fe1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:37.104 INFO 4486 --- [host-check-agent-status1,415526baea8672c8,064580c53141578d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:37.105 INFO 4486 --- [host-check-agent-status4,415526baea8672c8,cce73e791c99514a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:37.106 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:37.106 INFO 4486 --- [host-check-agent-status3,415526baea8672c8,ae96d7f4641d3fe1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:37.106 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:37.108 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:37.138 INFO 4486 --- [host-check-clockdiff1,2b6228a6cbbe1e59,80a753973361fd6d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056877 0 0 , error= 2024-10-16 13:34:37.140 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6af5c06f07d8d35e,c18eb06c8d7cc05a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6af5c06f07d8d35e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:37.140 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6af5c06f07d8d35e,c18eb06c8d7cc05a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6af5c06f07d8d35e, duration=25 ms] 2024-10-16 13:34:37.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:37.144 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:34:37.155 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.156 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:34:37.158 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:34:37.158 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:34:37.159 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:37.159 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.161 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:34:37.160511413+08:00 2024-10-16 13:34:37.161 INFO 4486 --- [host-check-agent-status3,415526baea8672c8,ae96d7f4641d3fe1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:34:37.161 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:34:37.161 INFO 4486 --- [host-check-agent-status2,415526baea8672c8,e01522ae085417bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:34:37.161 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:34:37.163844609+08:00 2024-10-16 13:34:37.161 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:34:37.161+08:00 2024-10-16 13:34:37.162 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:34:37.161+08:00 2024-10-16 13:34:37.162 INFO 4486 --- [host-check-agent-status1,415526baea8672c8,064580c53141578d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:34:37.163 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:34:37.164 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:34:37.165 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:34:37.161+08:00 and status AVAILABLE 2024-10-16 13:34:37.165 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:34:37.161+08:00 and status AVAILABLE 2024-10-16 13:34:37.166 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:34:37.168740761+08:00 2024-10-16 13:34:37.166 INFO 4486 --- [host-check-agent-status4,415526baea8672c8,cce73e791c99514a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:34:37.166 INFO 4486 --- [host-check-agent-status3,415526baea8672c8,ae96d7f4641d3fe1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:34:37.166 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:34:37.166+08:00 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-agent-status3,415526baea8672c8,ae96d7f4641d3fe1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-agent-status3,415526baea8672c8,ae96d7f4641d3fe1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-agent-status1,415526baea8672c8,064580c53141578d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-agent-status1,415526baea8672c8,064580c53141578d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-agent-status1,415526baea8672c8,064580c53141578d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:34:37.167 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:34:37.167 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.168 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:34:37.168 INFO 4486 --- [host-check-agent-status2,415526baea8672c8,e01522ae085417bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:34:37.168 INFO 4486 --- [host-check-agent-status2,415526baea8672c8,e01522ae085417bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:34:37.168 INFO 4486 --- [host-check-agent-status2,415526baea8672c8,e01522ae085417bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:34:37.169 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:34:37.166+08:00 and status AVAILABLE 2024-10-16 13:34:37.169 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.171 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.171 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:34:37.173748074+08:00 2024-10-16 13:34:37.171 INFO 4486 --- [host-check-agent-status4,415526baea8672c8,cce73e791c99514a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:34:37.171 INFO 4486 --- [host-check-agent-status4,415526baea8672c8,cce73e791c99514a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:34:37.171 INFO 4486 --- [host-check-agent-status4,415526baea8672c8,cce73e791c99514a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:34:37.172 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:34:37.171+08:00 2024-10-16 13:34:37.172 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:34:37.172 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.173 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:34:37.173 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:37.175 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:34:37.171+08:00 and status AVAILABLE 2024-10-16 13:34:37.176 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:37.177 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:37.177 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:34:37.177 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:37.179 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:37.181 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:37.183 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:37.183 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.185 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:37.186 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.186 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.186 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.186 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.186 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.186 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:37.187 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.189 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:37.197 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.198 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:34:37.199 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:37.200 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.201 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:34:37.203 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:34:37.204 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:37.206 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:37.211 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.211 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:34:37.212 INFO 4486 --- [ocp-schedules-20,dd870190ed6b4e72,99f993c96c65676e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:34:37.230 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:34:37.240 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:34:37.242 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:34:37.251 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:34:37.258 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:37.262 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:34:37.266 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:34:37.266 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:37.273 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:34:37.277 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:34:37.286 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:34:37.289 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:37.294 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:34:37.295 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:34:37.301 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:34:37.310 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:37.311 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:34:37.326 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:37.327 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:34:37.337 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:34:37.339 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:34:37.347 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:34:37.350 INFO 4486 --- [host-check-host-status3,72270bb42449cbc6,0086b2ad60dc3445] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:34:37.353 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:37.357 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:34:37.363 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:34:37.373 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:34:37.374 INFO 4486 --- [host-check-host-status4,72270bb42449cbc6,5cef9ac29b2bb973] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:34:37.385 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:34:37.397 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:37.397 INFO 4486 --- [host-check-host-status2,72270bb42449cbc6,ec1b2b1ff93882cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:34:37.399 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.399 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:34:37.400 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.401 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.402 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:34:37.412 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.412 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.412 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.412 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.412 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.413 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.431 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:34:37.439 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:34:37.442 INFO 4486 --- [host-check-host-status1,72270bb42449cbc6,6630ef9223389a31] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:34:37.482 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:34:37.522 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:37.536 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:37.538 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.539 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:34:37.540 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:34:37.544 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.545 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:34:37.546 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.546 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.547 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:34:37.555 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.555 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.555 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.555 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.555 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.556 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.563 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:34:37.754 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:34:37.797 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:37.801 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.802 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:34:37.803 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.803 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.804 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:34:37.813 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.813 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.813 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.813 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.813 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:37.813 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:37.831 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:34:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:34:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:34:37.994 INFO 4486 --- [prometheus-minute-schedule-1,70e2698f4549dad0,ede44cc30cbd3de4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:38.000 INFO 4486 --- [prometheus-minute-schedule-1,70e2698f4549dad0,ede44cc30cbd3de4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:38.026 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:34:38.069 INFO 4486 --- [host-check-clockdiff4,2b6228a6cbbe1e59,f90cad47c1dce402] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056878 3 3 , error= 2024-10-16 13:34:38.071 INFO 4486 --- [host-check-clockdiff2,2b6228a6cbbe1e59,3817c42fb2edb2d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056878 3 3 , error= 2024-10-16 13:34:38.096 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:34:38.097 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:38.098 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:34:38.099 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:38.100 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:38.101 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:34:38.111 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:38.111 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:38.111 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:38.111 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:38.111 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:34:38.111 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:38.178 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:34:38.396 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:34:38.181828, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:34:38.397 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:34:38.181828, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:34:38.436 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:34:38.181828} } 2024-10-16 13:34:38.502 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:34:38.511 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:34:38.514 INFO 4486 --- [ocp-schedules-23,cb19558bd8d649f4,e2a7f49d8e860f1f] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:34:38.873 INFO 4486 --- [ocp-schedules-9,94c111b8e425ee15,8be361dcb0f4eb4d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:34:38.908 INFO 4486 --- [host-check-clockdiff3,2b6228a6cbbe1e59,bf8e3a58076443ba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056878 3 3 , error= 2024-10-16 13:34:39.053 INFO 4486 --- [ocp-async-2,95665f95074b9a4f,7ddde782edb84f52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:39.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,82b5ba53f9a920e6,e689b6d12997ce9c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=82b5ba53f9a920e6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:39.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,82b5ba53f9a920e6,e689b6d12997ce9c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=82b5ba53f9a920e6, duration=2 ms] 2024-10-16 13:34:39.759 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:40.007 INFO 4486 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:34:40.007 INFO 4486 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:34:40.025 INFO 4486 --- [Check obproxy2,baea58cedd2adba4,d5336921a7ceae10] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:34:40.025 INFO 4486 --- [Check obproxy1,4428b3f2358c5214,54cd2ca939f5263c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:34:40.026 INFO 4486 --- [Check obproxy1,4428b3f2358c5214,54cd2ca939f5263c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:40.026 INFO 4486 --- [Check obproxy2,baea58cedd2adba4,d5336921a7ceae10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:40.030 INFO 4486 --- [Collect obproxy2,84534d72308ac187,fa85b36e1c8fdcc8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:34:40.030 INFO 4486 --- [Collect obproxy2,84534d72308ac187,fa85b36e1c8fdcc8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:40.032 INFO 4486 --- [Collect obproxy1,01a3db5dd19a4305,2814e4cf65337c05] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:34:40.032 INFO 4486 --- [Collect obproxy1,01a3db5dd19a4305,2814e4cf65337c05] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:34:40.049 INFO 4486 --- [Check obproxy2,baea58cedd2adba4,d5336921a7ceae10] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:34:40.049 INFO 4486 --- [Check obproxy2,baea58cedd2adba4,d5336921a7ceae10] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:34:40.050 INFO 4486 --- [Collect obproxy2,84534d72308ac187,fa85b36e1c8fdcc8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:34:40.054 INFO 4486 --- [Check obproxy1,4428b3f2358c5214,54cd2ca939f5263c] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:34:40.054 INFO 4486 --- [Check obproxy1,4428b3f2358c5214,54cd2ca939f5263c] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:34:40.055 INFO 4486 --- [Collect obproxy2,84534d72308ac187,fa85b36e1c8fdcc8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:34:40.060 INFO 4486 --- [Collect obproxy2,84534d72308ac187,fa85b36e1c8fdcc8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:34:40.063 INFO 4486 --- [Collect obproxy1,01a3db5dd19a4305,2814e4cf65337c05] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:34:40.069 INFO 4486 --- [Collect obproxy1,01a3db5dd19a4305,2814e4cf65337c05] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:34:40.074 INFO 4486 --- [Collect obproxy1,01a3db5dd19a4305,2814e4cf65337c05] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:34:40.337 INFO 4486 --- [ocp-schedules-1,01159865a7c1d97f,698b8a8c854356b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:34:40.863 INFO 4486 --- [ocp-schedules-21,3797b724f0e98de2,60d24979dff90ba9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:34:40.901 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8ebebcdead99c73c,99ace07056353575] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8ebebcdead99c73c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:40.901 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8ebebcdead99c73c,99ace07056353575] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8ebebcdead99c73c, duration=4 ms] 2024-10-16 13:34:40.937 WARN 4486 --- [scanner-13252192-1,3797b724f0e98de2,7d5aea84c251c03d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:34:40.974 INFO 4486 --- [alarm-detect-4,3797b724f0e98de2,8d408e1b07fc5107] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056855 ~ 1729056870 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056870, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056870, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056870=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:34:40.994 INFO 4486 --- [prometheus-minute-schedule-1,73843261e987d87a,6d6ebf1a90f8b481] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:40.995 INFO 4486 --- [prometheus-hour-schedule-1,2d5d48d2e24bc747,937d3e99da7900b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:34:41.007 INFO 4486 --- [prometheus-minute-schedule-1,73843261e987d87a,6d6ebf1a90f8b481] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:41.054 INFO 4486 --- [ocp-async-3,1b9d12d155324bfa,06154c2cc6f5893e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:41.260 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a9d4d7ff59e6a15e,9a910de3af8f275b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a9d4d7ff59e6a15e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:41.260 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a9d4d7ff59e6a15e,9a910de3af8f275b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a9d4d7ff59e6a15e, duration=1 ms] 2024-10-16 13:34:42.086 INFO 4486 --- [ocp-schedules-12,578484ed61da8fee,266ab869faae6139] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:34:42.150 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 13:34:42.150 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 13:34:42.152 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.153 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:34:42.154 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.155 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.156 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.156 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 13:34:42.178 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.179 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.179 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.180 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 13:34:42.195 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.195 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.196 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.197 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 13:34:42.211 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.212 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.213 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.214 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 13:34:42.228 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.228 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.229 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.229 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 13:34:42.236 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.237 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.237 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.238 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 13:34:42.243 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.243 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.244 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.244 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 13:34:42.249 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.249 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.250 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.251 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 13:34:42.257 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.258 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.258 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.259 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 13:34:42.265 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.265 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.266 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.266 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 13:34:42.273 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.273 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.274 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.275 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 13:34:42.284 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.285 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.286 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.286 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 13:34:42.301 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.302 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.303 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.303 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 13:34:42.308 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.309 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.309 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.310 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 13:34:42.316 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.317 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.317 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.318 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 13:34:42.325 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.325 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.326 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.326 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 13:34:42.331 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.332 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.332 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.332 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 13:34:42.337 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.338 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.338 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.339 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 13:34:42.343 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.344 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.344 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.345 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 13:34:42.350 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.351 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.351 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.352 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 13:34:42.357 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.357 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.358 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.358 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 13:34:42.364 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.364 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.365 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.365 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 13:34:42.370 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.370 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.371 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.371 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 13:34:42.376 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.376 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.377 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.377 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 13:34:42.382 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.383 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.383 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.384 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 13:34:42.388 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.389 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.389 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.390 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 13:34:42.395 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.395 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.396 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.396 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 13:34:42.401 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.401 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.402 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.403 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 13:34:42.407 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.407 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.408 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.408 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 13:34:42.413 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.413 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.414 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.414 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 13:34:42.418 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.419 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.419 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.420 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 13:34:42.426 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.426 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.427 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.427 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 13:34:42.433 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.433 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.434 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.434 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 13:34:42.440 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.440 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.441 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.441 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 13:34:42.451 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.451 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.452 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.452 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 13:34:42.460 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.460 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.461 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.461 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 13:34:42.467 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.468 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.468 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.469 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 13:34:42.474 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.474 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.475 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.475 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 13:34:42.479 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.480 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.480 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.481 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 13:34:42.485 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.486 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.486 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.487 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 13:34:42.491 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.492 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.493 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.493 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 13:34:42.498 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.498 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.499 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.499 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 13:34:42.513 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.514 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.515 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.515 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 13:34:42.537 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.538 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.538 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.539 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 13:34:42.546 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.547 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.547 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.548 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 13:34:42.554 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.555 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.556 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.556 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 13:34:42.563 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.564 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.564 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.565 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 13:34:42.569 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.570 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.570 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.571 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 13:34:42.576 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.576 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.577 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.577 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 13:34:42.582 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 13:34:42.582 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 13:34:42.584 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.585 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:34:42.586 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.586 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.587 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.588 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 13:34:42.595 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.595 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:34:42.596 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:34:42.597 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 13:34:42.602 INFO 4486 --- [ocp-schedules-23,2568584a5e8e39de,747e90c993a3ff8c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 13:34:42.760 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:42.989 INFO 4486 --- [ocp-async-9,d7e62af518dc9df8,c826fb26b0c5b409] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:34:43.054 INFO 4486 --- [ocp-async-4,b25b64c257adb687,93489ad43170c493] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:43.994 INFO 4486 --- [prometheus-minute-schedule-1,b0deb3c5209b51d1,71a1e23319cfff23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:44.001 INFO 4486 --- [prometheus-minute-schedule-1,b0deb3c5209b51d1,71a1e23319cfff23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:45.054 INFO 4486 --- [ocp-async-5,a25af06a3167bb58,a6dd0a80ee6fadaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:45.449 INFO 4486 --- [ocp-schedules-2,2ae17d2ae3581f15,55873945d53e7312] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:34:45.454 INFO 4486 --- [ocp-schedules-2,2ae17d2ae3581f15,55873945d53e7312] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:34:45.454 INFO 4486 --- [ocp-schedules-2,2ae17d2ae3581f15,55873945d53e7312] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:34:45.469 INFO 4486 --- [ocp-schedules-28,208cb4b4571c0fa6,572aa82cc859254a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:34:45.469 INFO 4486 --- [ocp-schedules-28,208cb4b4571c0fa6,572aa82cc859254a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:34:45.761 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:46.994 INFO 4486 --- [prometheus-minute-schedule-1,598353b2843e3feb,fe15f61ec8f54b3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:47.001 INFO 4486 --- [prometheus-minute-schedule-1,598353b2843e3feb,fe15f61ec8f54b3e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:47.054 INFO 4486 --- [ocp-async-3,b4827e6ad695d275,ccb77d427501236f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:47.070 INFO 4486 --- [ocp-schedules-24,b186d762d344c687,8c153f925397b103] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:34:47.096 INFO 4486 --- [ocp-schedules-11,9509c43d37978f93,db8e7ade91624da3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:34:47.148 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,16d914d73e4c9b3f,05224e444cc0890c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=16d914d73e4c9b3f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:47.148 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,16d914d73e4c9b3f,05224e444cc0890c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=16d914d73e4c9b3f, duration=3 ms] 2024-10-16 13:34:47.162 INFO 4486 --- [ocp-schedules-4,633e7c636a764ed2,e48d5904caec30ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:34:48.761 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:49.055 INFO 4486 --- [ocp-async-8,d05417f91a7744cd,f78cbb846acfbede] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:49.328 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1e9b81a7d3b5e5ad,0a07a09fa35ee8ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1e9b81a7d3b5e5ad, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:49.328 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1e9b81a7d3b5e5ad,0a07a09fa35ee8ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1e9b81a7d3b5e5ad, duration=1 ms] 2024-10-16 13:34:49.994 INFO 4486 --- [prometheus-minute-schedule-1,b7baba47b6cc31ef,8f953b39819bdf4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:50.002 INFO 4486 --- [prometheus-minute-schedule-1,b7baba47b6cc31ef,8f953b39819bdf4d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:50.026 INFO 4486 --- [ocp-schedules-25,4527967d7919eb8d,78dbdadbe4359f20] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:34:50.026 INFO 4486 --- [ocp-schedules-25,4527967d7919eb8d,78dbdadbe4359f20] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:34:50.027 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:34:50.027 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:34:50.027 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:34:50.027 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:34:50.027 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:34:50.033 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:34:50.033 INFO 4486 --- [ocp-schedules-25,4527967d7919eb8d,78dbdadbe4359f20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:34:50.033 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:34:50.033 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:34:50.033 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:34:50.034 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:34:50.035 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:34:50.035 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:34:50.035 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:34:50.035 INFO 4486 --- [ocp-schedules-25,4527967d7919eb8d,78dbdadbe4359f20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:34:50.035 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:34:50.035 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:34:50.066 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:34:50.067 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:34:50.071 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:34:50.072 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:34:50.072 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:34:50.074 INFO 4486 --- [ocp-schedules-25,4527967d7919eb8d,78dbdadbe4359f20] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:34:50.083 INFO 4486 --- [ocp-schedules-25,4527967d7919eb8d,78dbdadbe4359f20] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:34:50.343 INFO 4486 --- [ocp-schedules-14,30eab9b952d72a77,0480ec2a2542589e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:34:50.377 INFO 4486 --- [ocp-schedules-13,e5dc31e78db8f5ab,90307f5b1f44f90c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:34:50.377 INFO 4486 --- [ocp-schedules-13,e5dc31e78db8f5ab,90307f5b1f44f90c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:34:50.379 INFO 4486 --- [ocp-schedules-13,e5dc31e78db8f5ab,90307f5b1f44f90c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:34:50.379 INFO 4486 --- [ocp-schedules-13,e5dc31e78db8f5ab,90307f5b1f44f90c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:34:50.380 INFO 4486 --- [ocp-schedules-13,e5dc31e78db8f5ab,90307f5b1f44f90c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:34:50.380 INFO 4486 --- [ocp-schedules-13,e5dc31e78db8f5ab,90307f5b1f44f90c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:34:50.578 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,859178e375f68ddb,99291d9bd2ba82d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=859178e375f68ddb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:34:50.579 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,859178e375f68ddb,99291d9bd2ba82d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=859178e375f68ddb, duration=1 ms] 2024-10-16 13:34:50.908 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9a2ceb644ecda737,94cffbf6ef9ee0a3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9a2ceb644ecda737, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:50.908 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9a2ceb644ecda737,94cffbf6ef9ee0a3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9a2ceb644ecda737, duration=1 ms] 2024-10-16 13:34:51.046 INFO 4486 --- [ocp-schedules-5,84a15ff79bee5a3f,42b666775d1ad97e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:34:51.055 INFO 4486 --- [ocp-async-2,25cca67aff30823b,e90e0d9a1c21418b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:51.081 WARN 4486 --- [scanner-13252412-1,84a15ff79bee5a3f,32f3ac7ec0bb228e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:34:51.265 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b6c7da7686108245,b5be7432d2775e70] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b6c7da7686108245, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:51.266 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b6c7da7686108245,b5be7432d2775e70] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b6c7da7686108245, duration=2 ms] 2024-10-16 13:34:51.762 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:52.989 INFO 4486 --- [ocp-async-1,516a184c4e56397e,7cb858aaf49086ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:34:52.994 INFO 4486 --- [prometheus-minute-schedule-1,1c7bc4d1cd13abee,c7263cd1aa64b82a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:53.000 INFO 4486 --- [prometheus-minute-schedule-1,1c7bc4d1cd13abee,c7263cd1aa64b82a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:53.055 INFO 4486 --- [ocp-async-3,b6f94dc35e03ac18,497b6eed49ef8564] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:54.763 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:55.056 INFO 4486 --- [ocp-async-4,74ababd58bb6e6b9,2022fa36c2aca996] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:55.994 INFO 4486 --- [prometheus-minute-schedule-1,22669435d061e3dc,5ef0f83989f9d87c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:56.001 INFO 4486 --- [prometheus-minute-schedule-1,22669435d061e3dc,5ef0f83989f9d87c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:57.056 INFO 4486 --- [ocp-async-5,3979dd06156a782c,4dd604c5e5be2c88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:57.167 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0aed43201f5d7a88,7d3dff4be784090b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0aed43201f5d7a88, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:57.168 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0aed43201f5d7a88,7d3dff4be784090b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0aed43201f5d7a88, duration=3 ms] 2024-10-16 13:34:57.764 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:34:58.878 INFO 4486 --- [ocp-schedules-10,b5ae4ad2d56b2c62,b9b1dde41c00b64d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:34:58.994 INFO 4486 --- [prometheus-minute-schedule-1,ce3475e7afe0cda5,818d2933867019ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056900, interval=60 2024-10-16 13:34:59.001 INFO 4486 --- [prometheus-minute-schedule-1,ce3475e7afe0cda5,818d2933867019ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:34:59.056 INFO 4486 --- [ocp-async-1,c37c8f11df573d02,578890c442336fa2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:34:59.333 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a5c0fedfeb89d5f3,54e6669a6c4c79e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a5c0fedfeb89d5f3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:34:59.333 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a5c0fedfeb89d5f3,54e6669a6c4c79e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a5c0fedfeb89d5f3, duration=1 ms] 2024-10-16 13:35:00.348 INFO 4486 --- [ocp-schedules-29,a122726b8a49e6da,31a9bd7cb4653fbc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:35:00.699 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:35:00.699 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:35:00.707 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:35:00.707 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:35:00.707 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:35:00.707 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:35:00.712 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:35:00.712 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:35:00.712 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:35:00.712 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:35:00.759 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:35:00.765 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:00.766 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:35:00.766 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:35:00.766 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:35:00.766 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:35:00.766 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:35:00.773 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:35:00.919 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,942f20d4013384d0,3aebac846ed444f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=942f20d4013384d0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:00.920 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,942f20d4013384d0,3aebac846ed444f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=942f20d4013384d0, duration=2 ms] 2024-10-16 13:35:01.056 INFO 4486 --- [ocp-async-3,6696e252074f9692,c827af0c0f350a5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:01.169 INFO 4486 --- [ocp-schedules-18,337a33bf4efee6cb,07015c4efd9541b9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:35:01.169+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:35:01.173 INFO 4486 --- [ocp-schedules-18,337a33bf4efee6cb,07015c4efd9541b9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:35:01.177 INFO 4486 --- [ocp-schedules-18,337a33bf4efee6cb,07015c4efd9541b9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:35:01.188 INFO 4486 --- [ocp-schedules-18,337a33bf4efee6cb,07015c4efd9541b9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:35:01.253 INFO 4486 --- [ocp-schedules-21,dd52141da4b565a7,d99201533cff1c5c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:35:01.271 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,db9086aa48aeac4f,987e563706c39920] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=db9086aa48aeac4f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:01.271 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,db9086aa48aeac4f,987e563706c39920] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=db9086aa48aeac4f, duration=1 ms] 2024-10-16 13:35:01.277 WARN 4486 --- [scanner-13252729-1,dd52141da4b565a7,0023c37a149488d5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:35:01.279 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:35:01.329 INFO 4486 --- [alarm-detect-7,dd52141da4b565a7,024505e23abbc6fc] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056870 ~ 1729056891 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056885, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056885, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056885=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:35:01.331 INFO 4486 --- [alarm-detect-7,dd52141da4b565a7,024505e23abbc6fc] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:35:01.710 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.998365542S 2024-10-16 13:35:01.711 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:35:01.778 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:35:01.778 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:35:01.806 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:35:01.876 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:35:01.876 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:35:01.876 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:35:01.876 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:35:01.876 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:35:01.876 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:35:01.884 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:35:01.994 INFO 4486 --- [prometheus-minute-schedule-1,5688041535ac0769,7c39f6485e9ed2c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:02.000 INFO 4486 --- [prometheus-minute-schedule-1,5688041535ac0769,7c39f6485e9ed2c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:02.016 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] o.s.boot.SpringApplication : Started application in 1.309 seconds (JVM running for 77805.067) 2024-10-16 13:35:02.016 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.309721304S 2024-10-16 13:35:02.045 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ef1d159215c3a513,9e45e8d62f150fd0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=ef1d159215c3a513, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 13:35:02.244 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ef1d159215c3a513,9e45e8d62f150fd0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=ef1d159215c3a513, duration=210 ms] 2024-10-16 13:35:02.395 INFO 4486 --- [ocp-schedules-9,e6c21fa0a64af4b1,2c69bb6ab273bd9e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:35:02.540 INFO 4486 --- [ocp-schedules-16,60111c7cfacea21e,ddff8b96e94d7c82] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:35:02.540 INFO 4486 --- [ocp-schedules-16,833437b29be7ad37,9d911577d9ac3089] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:35:02.541 INFO 4486 --- [ocp-schedules-16,833437b29be7ad37,9d911577d9ac3089] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:02.589 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.590 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:02.592 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.593 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:02.606 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.607 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:02.624 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:02.628 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.629 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:35:02.655 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:02.656 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.656 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:02.657 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:35:02.661 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.664 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:35:02.665 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:02.679 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.679 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:02.695 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.701 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:02.702 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.703 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:35:02.704 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:02.705 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.705 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:02.706 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:35:02.710 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:35:02.714 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:02.716 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:02.740 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:35:02.744 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:35:02.744 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:35:02.757 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:02.759 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.760 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:02.764 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.768 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:35:02.771 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:02.800 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.801 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:02.821 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:35:02.833 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:02.835 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.835 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:02.837 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.838 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:35:02.848 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:35:02.849 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:35:02.863 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.863 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:02.890 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.891 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:02.917 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.917 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:02.947 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:02.974 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:02.975 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:02.976 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:02.980 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:35:02.985 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:35:02.990 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:35:02.990 INFO 4486 --- [ocp-async-3,9b8c686f8b2518f2,a8921d76904e7c4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:35:02.992 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:35:03.019 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:35:03.057 INFO 4486 --- [ocp-async-4,0d19e51a192aff88,91c1a512a7292df1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:03.153 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:35:03.187 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:03.188 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:03.189 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:35:03.224 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:35:03.224 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:35:03.766 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:04.994 INFO 4486 --- [prometheus-minute-schedule-1,ca1399c3f8fadc47,688efab45f43e23c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:05.000 INFO 4486 --- [prometheus-minute-schedule-1,ca1399c3f8fadc47,688efab45f43e23c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:05.057 INFO 4486 --- [ocp-async-7,d0d8606bf8628f49,e355d3075960cd78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:06.767 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:06.856 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,50164a1c2561dd9d,1cb2fcc044dc19a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=50164a1c2561dd9d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:35:06.857 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,50164a1c2561dd9d,1cb2fcc044dc19a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=50164a1c2561dd9d, duration=3 ms] 2024-10-16 13:35:07.057 INFO 4486 --- [ocp-async-9,45b3e10054a6f618,61a2f73bbd76743d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:07.066 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:35:07.067 INFO 4486 --- [ocp-schedules-4,55e2d1b2d98442d7,b9e05ed5994a2941] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:35:07.069 INFO 4486 --- [ocp-schedules-4,55e2d1b2d98442d7,b9e05ed5994a2941] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:07.081 INFO 4486 --- [ocp-schedules-14,63caa6ad40f3bb4a,efae7cf210bed631] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:35:07.083 INFO 4486 --- [ocp-schedules-14,63caa6ad40f3bb4a,efae7cf210bed631] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:35:07.089 INFO 4486 --- [ocp-schedules-18,d4f514cae4181838,eaae74288a841048] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:35:07.090 INFO 4486 --- [host-check-clockdiff1,d4f514cae4181838,dccc90df530a7b96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:35:07.090 INFO 4486 --- [host-check-clockdiff2,d4f514cae4181838,1c1522556536b13b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:35:07.091 INFO 4486 --- [host-check-clockdiff3,d4f514cae4181838,a893467e0cf7b0b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:35:07.091 INFO 4486 --- [host-check-clockdiff4,d4f514cae4181838,46b4d83b002c6900] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:35:07.092 INFO 4486 --- [ocp-schedules-22,82ae2a5c867498a1,a3f190cd73007dfa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:35:07.092 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:35:07.092 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:35:07.094 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:35:07.093 INFO 4486 --- [ocp-schedules-3,89b894deed3d10c3,49ab343a959e3de6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:35:07.094 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:35:07.095 INFO 4486 --- [host-check-agent-status1,89b894deed3d10c3,1df6629356bfa2d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:35:07.096 INFO 4486 --- [host-check-agent-status3,89b894deed3d10c3,e4f7b41101dd1101] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:35:07.096 INFO 4486 --- [host-check-agent-status4,89b894deed3d10c3,d06b572374393f08] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:35:07.096 INFO 4486 --- [host-check-agent-status2,89b894deed3d10c3,72ccbca492e5b8a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:35:07.104 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:07.121 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:35:07.123 INFO 4486 --- [host-check-clockdiff1,d4f514cae4181838,dccc90df530a7b96] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056907 0 0 , error= 2024-10-16 13:35:07.124 INFO 4486 --- [host-check-agent-status1,89b894deed3d10c3,1df6629356bfa2d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:07.124 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:35:07.124 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:35:07.125 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:07.125 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:35:07.126 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:35:07.126 INFO 4486 --- [host-check-agent-status2,89b894deed3d10c3,72ccbca492e5b8a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:35:07.126 INFO 4486 --- [host-check-agent-status4,89b894deed3d10c3,d06b572374393f08] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:35:07.126 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:35:07.126 INFO 4486 --- [host-check-agent-status1,89b894deed3d10c3,1df6629356bfa2d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:07.127 INFO 4486 --- [host-check-agent-status3,89b894deed3d10c3,e4f7b41101dd1101] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:35:07.128 INFO 4486 --- [host-check-agent-status4,89b894deed3d10c3,d06b572374393f08] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:35:07.128 INFO 4486 --- [host-check-agent-status2,89b894deed3d10c3,72ccbca492e5b8a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:35:07.128 INFO 4486 --- [host-check-agent-status3,89b894deed3d10c3,e4f7b41101dd1101] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:35:07.130 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.131 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:07.132 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:35:07.153 INFO 4486 --- [host-check-agent-status1,89b894deed3d10c3,1df6629356bfa2d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:35:07.153 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:35:07.154 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:35:07.155 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:35:07.155 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:35:07.155303207+08:00 2024-10-16 13:35:07.155 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.157 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:35:07.156+08:00 2024-10-16 13:35:07.157 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:07.157 INFO 4486 --- [host-check-agent-status2,89b894deed3d10c3,72ccbca492e5b8a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:35:07.157 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:35:07.159966717+08:00 2024-10-16 13:35:07.158 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:35:07.160454778+08:00 2024-10-16 13:35:07.158 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:35:07.158+08:00 2024-10-16 13:35:07.159 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:35:07.158+08:00 2024-10-16 13:35:07.159 INFO 4486 --- [host-check-agent-status4,89b894deed3d10c3,d06b572374393f08] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:35:07.161 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:35:07.161 INFO 4486 --- [host-check-agent-status1,89b894deed3d10c3,1df6629356bfa2d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:35:07.162 INFO 4486 --- [host-check-agent-status1,89b894deed3d10c3,1df6629356bfa2d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:35:07.162 INFO 4486 --- [host-check-agent-status1,89b894deed3d10c3,1df6629356bfa2d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:35:07.162 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:35:07.162 INFO 4486 --- [host-check-agent-status3,89b894deed3d10c3,e4f7b41101dd1101] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:35:07.163 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:35:07.166508308+08:00 2024-10-16 13:35:07.164 INFO 4486 --- [host-check-agent-status2,89b894deed3d10c3,72ccbca492e5b8a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:35:07.164 INFO 4486 --- [host-check-agent-status2,89b894deed3d10c3,72ccbca492e5b8a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:35:07.164 INFO 4486 --- [host-check-agent-status2,89b894deed3d10c3,72ccbca492e5b8a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:35:07.164 INFO 4486 --- [host-check-agent-status4,89b894deed3d10c3,d06b572374393f08] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:35:07.164 INFO 4486 --- [host-check-agent-status4,89b894deed3d10c3,d06b572374393f08] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:35:07.164 INFO 4486 --- [host-check-agent-status4,89b894deed3d10c3,d06b572374393f08] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:35:07.166 INFO 4486 --- [host-check-agent-status3,89b894deed3d10c3,e4f7b41101dd1101] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:35:07.166 INFO 4486 --- [host-check-agent-status3,89b894deed3d10c3,e4f7b41101dd1101] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:35:07.166 INFO 4486 --- [host-check-agent-status3,89b894deed3d10c3,e4f7b41101dd1101] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:35:07.166 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:35:07.164+08:00 2024-10-16 13:35:07.170 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:35:07.156+08:00 and status AVAILABLE 2024-10-16 13:35:07.172 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.172 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:35:07.158+08:00 and status AVAILABLE 2024-10-16 13:35:07.173 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:35:07.173 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:35:07.158+08:00 and status AVAILABLE 2024-10-16 13:35:07.174 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,95daaf62de67a5de,2c7346419528ed1c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=95daaf62de67a5de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:07.174 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,95daaf62de67a5de,2c7346419528ed1c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=95daaf62de67a5de, duration=1 ms] 2024-10-16 13:35:07.174 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:35:07.175 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:35:07.176 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:35:07.164+08:00 and status AVAILABLE 2024-10-16 13:35:07.177 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:35:07.178 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:35:07.182 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:07.183 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:35:07.183 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:35:07.184 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:35:07.184 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:07.185 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:35:07.187 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:35:07.189 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:35:07.208 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:07.211 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:07.214 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:07.216 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:07.240 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:35:07.249 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:35:07.252 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:35:07.252 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:35:07.260 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:35:07.262 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:35:07.263 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:35:07.272 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:35:07.276 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:07.277 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:07.279 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:07.287 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:07.304 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:35:07.314 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:35:07.315 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:35:07.316 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.316 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:35:07.317 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:35:07.326 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:35:07.331 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.331 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:35:07.332 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:35:07.334 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:07.342 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:07.342 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:35:07.345 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.346 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:35:07.347 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:07.348 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.349 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:35:07.354 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:35:07.364 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:07.365 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:35:07.366 INFO 4486 --- [ocp-schedules-27,62982e9f1dbd9b50,3bc132079e8ce559] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:35:07.370 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:07.376 INFO 4486 --- [host-check-host-status3,82ae2a5c867498a1,7cd29a198e8e8026] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:35:07.382 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:35:07.391 INFO 4486 --- [host-check-host-status4,82ae2a5c867498a1,5e4b2f3b9614f56a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:35:07.397 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:35:07.409 INFO 4486 --- [host-check-host-status2,82ae2a5c867498a1,9fd596fd1d1146b1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:35:07.431 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:35:07.441 INFO 4486 --- [host-check-host-status1,82ae2a5c867498a1,d34bca6b8a9ffcf8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:35:07.994 INFO 4486 --- [prometheus-minute-schedule-1,276c9e1ec77f3a53,856dd9c2257983b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:08.000 INFO 4486 --- [prometheus-minute-schedule-1,276c9e1ec77f3a53,856dd9c2257983b4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:08.054 INFO 4486 --- [host-check-clockdiff4,d4f514cae4181838,46b4d83b002c6900] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056908 2 2 , error= 2024-10-16 13:35:08.056 INFO 4486 --- [host-check-clockdiff3,d4f514cae4181838,a893467e0cf7b0b3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056908 3 3 , error= 2024-10-16 13:35:08.056 INFO 4486 --- [host-check-clockdiff2,d4f514cae4181838,1c1522556536b13b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056908 3 3 , error= 2024-10-16 13:35:09.058 INFO 4486 --- [ocp-async-3,254cbf8e270a9887,ccd8fe521230617a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:09.339 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d2d736222e718811,f91d34a19a10a107] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d2d736222e718811, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:09.339 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d2d736222e718811,f91d34a19a10a107] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d2d736222e718811, duration=2 ms] 2024-10-16 13:35:09.767 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:10.007 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:35:10.010 INFO 4486 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:35:10.027 INFO 4486 --- [Check obproxy1,ef4b392cbdc4ace1,d088bc3a4cb2b403] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:35:10.030 INFO 4486 --- [Check obproxy2,f99d38be6b6b768b,c725e7f0525d3995] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:35:10.032 INFO 4486 --- [Check obproxy1,ef4b392cbdc4ace1,d088bc3a4cb2b403] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:10.035 INFO 4486 --- [Check obproxy2,f99d38be6b6b768b,c725e7f0525d3995] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:10.035 INFO 4486 --- [Collect obproxy2,7c5be88466114dd9,ba9bb4b473edc2bc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:35:10.035 INFO 4486 --- [Collect obproxy2,7c5be88466114dd9,ba9bb4b473edc2bc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:10.035 INFO 4486 --- [Collect obproxy1,f99d792b391e2e00,7f40843a59f347a4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:35:10.035 INFO 4486 --- [Collect obproxy1,f99d792b391e2e00,7f40843a59f347a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:10.055 INFO 4486 --- [Check obproxy1,ef4b392cbdc4ace1,d088bc3a4cb2b403] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:35:10.055 INFO 4486 --- [Check obproxy1,ef4b392cbdc4ace1,d088bc3a4cb2b403] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:35:10.057 INFO 4486 --- [Collect obproxy2,7c5be88466114dd9,ba9bb4b473edc2bc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:35:10.060 INFO 4486 --- [Check obproxy2,f99d38be6b6b768b,c725e7f0525d3995] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:35:10.060 INFO 4486 --- [Check obproxy2,f99d38be6b6b768b,c725e7f0525d3995] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:35:10.063 INFO 4486 --- [Collect obproxy2,7c5be88466114dd9,ba9bb4b473edc2bc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:35:10.066 INFO 4486 --- [Collect obproxy1,f99d792b391e2e00,7f40843a59f347a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:35:10.067 INFO 4486 --- [Collect obproxy2,7c5be88466114dd9,ba9bb4b473edc2bc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:35:10.069 INFO 4486 --- [Collect obproxy1,f99d792b391e2e00,7f40843a59f347a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:35:10.072 INFO 4486 --- [Collect obproxy1,f99d792b391e2e00,7f40843a59f347a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:35:10.354 INFO 4486 --- [ocp-schedules-12,ceef0a7c06a51b48,f09f80d6a1f57e66] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:35:10.924 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,95f06fb73a557e48,2011d5e9e684c845] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=95f06fb73a557e48, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:10.924 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,95f06fb73a557e48,2011d5e9e684c845] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=95f06fb73a557e48, duration=1 ms] 2024-10-16 13:35:10.979 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:35:10.994 INFO 4486 --- [prometheus-minute-schedule-1,c6b87aa5d0936a0f,05181a073753eaf4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:11.000 INFO 4486 --- [prometheus-minute-schedule-1,c6b87aa5d0936a0f,05181a073753eaf4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:35:11.058 INFO 4486 --- [ocp-async-5,eef9a8927040652a,613d419b0939940c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:11.276 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,5a420afb17f1bf83,076fc309bb0842e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5a420afb17f1bf83, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:11.276 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,5a420afb17f1bf83,076fc309bb0842e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5a420afb17f1bf83, duration=2 ms] 2024-10-16 13:35:11.409 INFO 4486 --- [ocp-schedules-25,2e90b288a31fe71b,d86dbdf4263cfd9a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:35:11.432 WARN 4486 --- [scanner-13253771-1,2e90b288a31fe71b,a63d7c5975276b55] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:35:11.506 INFO 4486 --- [alarm-detect-10,2e90b288a31fe71b,fd43af11195c47d2] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056885 ~ 1729056901 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056900, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056900, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056900=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:35:11.638 INFO 4486 --- [ocp-schedules-12,26484e6432dea109,1666f381e8ef8004] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 13:35:11.638 INFO 4486 --- [ocp-schedules-12,26484e6432dea109,1666f381e8ef8004] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 13:35:11.671 INFO 4486 --- [pool-278-thread-1,26484e6432dea109,bec59e632eb91e3c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:35:11.672 INFO 4486 --- [pool-278-thread-2,26484e6432dea109,de75aaefc8971767] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:35:11.672 INFO 4486 --- [pool-278-thread-3,26484e6432dea109,403df3f6ea0ddcbf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:35:11.690 INFO 4486 --- [pool-278-thread-1,26484e6432dea109,bec59e632eb91e3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:11.693 INFO 4486 --- [pool-278-thread-2,26484e6432dea109,de75aaefc8971767] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:11.694 INFO 4486 --- [pool-278-thread-1,26484e6432dea109,bec59e632eb91e3c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056911000000] 2024-10-16 13:35:11.697 INFO 4486 --- [pool-278-thread-2,26484e6432dea109,de75aaefc8971767] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056911000000] 2024-10-16 13:35:11.708 INFO 4486 --- [pool-278-thread-3,26484e6432dea109,403df3f6ea0ddcbf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:11.712 INFO 4486 --- [pool-278-thread-3,26484e6432dea109,403df3f6ea0ddcbf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729056911000000] 2024-10-16 13:35:11.986 WARN 4486 --- [pool-278-thread-3,26484e6432dea109,403df3f6ea0ddcbf] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 13:35:11.986 WARN 4486 --- [pool-278-thread-3,26484e6432dea109,403df3f6ea0ddcbf] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 13:35:11.988 INFO 4486 --- [pool-278-thread-3,26484e6432dea109,403df3f6ea0ddcbf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:35:12.295 WARN 4486 --- [pool-278-thread-1,26484e6432dea109,bec59e632eb91e3c] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 13:35:12.297 INFO 4486 --- [pool-278-thread-1,26484e6432dea109,bec59e632eb91e3c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:35:12.510 WARN 4486 --- [pool-278-thread-2,26484e6432dea109,de75aaefc8971767] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 13:35:12.512 INFO 4486 --- [pool-278-thread-2,26484e6432dea109,de75aaefc8971767] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:35:12.513 INFO 4486 --- [ocp-schedules-12,26484e6432dea109,1666f381e8ef8004] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 13:35:12.768 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:12.990 INFO 4486 --- [ocp-async-7,a7f4795ff4348620,917381cff580c01f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:35:13.058 INFO 4486 --- [ocp-async-8,9603e43863bb1d1e,e76182af924bd21b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:13.097 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0a585100dc20614a,44319a0ff589b05a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=0a585100dc20614a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:35:13.098 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0a585100dc20614a,44319a0ff589b05a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=0a585100dc20614a, duration=2 ms] 2024-10-16 13:35:13.222 ERROR 4486 --- [metric-archive-second-11,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=4117402) Duplicate entry '587073990944165888-1729056840' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:35:13.221114] [YB420AC531B8-00060B7DB7688406-0-0]; nested exception is java.sql.BatchUpdateException: (conn=4117402) Duplicate entry '587073990944165888-1729056840' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:35:13.221114] [YB420AC531B8-00060B7DB7688406-0-0] 2024-10-16 13:35:13.224 ERROR 4486 --- [metric-archive-second-26,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=70525) Duplicate entry '586679390194700288-1729056840' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:35:13.223228] [YB420AC531B8-00060B7D84EAACCD-0-0]; nested exception is java.sql.BatchUpdateException: (conn=70525) Duplicate entry '586679390194700288-1729056840' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:35:13.223228] [YB420AC531B8-00060B7D84EAACCD-0-0] 2024-10-16 13:35:13.245 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056840, nodeCount=1000, affectRows=1000 2024-10-16 13:35:13.254 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056840, nodeCount=998, validCount=998 2024-10-16 13:35:13.260 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056840, nodeCount=1000, affectRows=1000 2024-10-16 13:35:13.265 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056840, nodeCount=1000, validCount=1000 2024-10-16 13:35:13.566 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056840, nodeCount=571, affectRows=571 2024-10-16 13:35:13.568 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056840, nodeCount=571, validCount=571 2024-10-16 13:35:13.705 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056840, nodeCount=878, affectRows=878 2024-10-16 13:35:13.709 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056840, nodeCount=877, validCount=877 2024-10-16 13:35:13.994 INFO 4486 --- [prometheus-minute-schedule-1,867883281619f936,25c2c5757fca4141] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:14.002 INFO 4486 --- [prometheus-minute-schedule-1,867883281619f936,25c2c5757fca4141] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:14.397 INFO 4486 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056840, nodeCount=132, affectRows=132 2024-10-16 13:35:14.399 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056840, nodeCount=184, affectRows=184 2024-10-16 13:35:14.399 INFO 4486 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056840, nodeCount=164, affectRows=164 2024-10-16 13:35:14.581 INFO 4486 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056840, nodeCount=132, validCount=132 2024-10-16 13:35:14.632 INFO 4486 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056840, nodeCount=164, validCount=164 2024-10-16 13:35:14.644 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056840, nodeCount=184, validCount=184 2024-10-16 13:35:15.074 INFO 4486 --- [ocp-async-2,cad490667506d024,e33d46490ea2e1fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:15.769 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:16.994 INFO 4486 --- [prometheus-minute-schedule-1,26d40eaf25f867e9,2e33386d2de12615] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:17.000 INFO 4486 --- [prometheus-minute-schedule-1,26d40eaf25f867e9,2e33386d2de12615] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:17.070 INFO 4486 --- [ocp-schedules-4,55e2d1b2d98442d7,b9e05ed5994a2941] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:35:17.075 INFO 4486 --- [ocp-async-5,61f9707bd8c79bc0,2a5f07ae81c6f937] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:17.099 INFO 4486 --- [ocp-schedules-19,077653c47d7ebdaa,473bdb71212673a1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:35:17.162 INFO 4486 --- [ocp-schedules-14,63caa6ad40f3bb4a,efae7cf210bed631] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:35:17.178 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,474fdd5d2107e752,a23f31c3c577f51b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=474fdd5d2107e752, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:17.178 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,474fdd5d2107e752,a23f31c3c577f51b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=474fdd5d2107e752, duration=1 ms] 2024-10-16 13:35:18.771 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:18.884 INFO 4486 --- [ocp-schedules-17,c56c820d0f245f37,65ed464716c7692b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:35:19.075 INFO 4486 --- [ocp-async-9,0faa7650241a4347,c32ffaae1ac03e72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:19.343 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,66309f12c2c9cb30,4f0da83ceb9ea7cd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=66309f12c2c9cb30, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:19.343 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,66309f12c2c9cb30,4f0da83ceb9ea7cd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=66309f12c2c9cb30, duration=1 ms] 2024-10-16 13:35:19.994 INFO 4486 --- [prometheus-minute-schedule-1,d806ba7fef32571a,457f7519e7d52d93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:19.999 INFO 4486 --- [prometheus-minute-schedule-1,d806ba7fef32571a,457f7519e7d52d93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:20.360 INFO 4486 --- [ocp-schedules-30,23171b787e6364ab,3ba29ffd0cc79bea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:35:20.928 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9039480f342984b0,62d64f816d8b5bc3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9039480f342984b0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:20.929 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9039480f342984b0,62d64f816d8b5bc3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9039480f342984b0, duration=2 ms] 2024-10-16 13:35:20.996 INFO 4486 --- [prometheus-cache-monitor1,b9834b825976677b,5c9eda1f25b3a57b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:35:20.997 INFO 4486 --- [prometheus-cache-monitor1,b9834b825976677b,5c9eda1f25b3a57b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:35:21.075 INFO 4486 --- [ocp-async-3,aff48d2b78fef926,76eceac59d7fa077] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:21.281 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,085ba2679e7cdaec,8b655b1043893c1f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=085ba2679e7cdaec, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:21.281 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,085ba2679e7cdaec,8b655b1043893c1f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=085ba2679e7cdaec, duration=1 ms] 2024-10-16 13:35:21.586 INFO 4486 --- [ocp-schedules-2,f7ca8a5425ce4754,6cb6ea8c281b9f83] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:35:21.638 WARN 4486 --- [scanner-13254138-1,f7ca8a5425ce4754,9bf26fd02006ad85] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:35:21.772 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:22.799 INFO 4486 --- [ocp-schedules-21,dd34405c98d7bf3f,e4a6dee660df6d07] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:35:22.990 INFO 4486 --- [ocp-async-2,56cc701ea627ec10,789ebda2932240c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:35:22.994 INFO 4486 --- [prometheus-minute-schedule-1,8914f00046ef7ed4,1544f7ab32bea20c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:23.000 INFO 4486 --- [prometheus-minute-schedule-1,8914f00046ef7ed4,1544f7ab32bea20c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:23.075 INFO 4486 --- [ocp-async-5,72d077ebb4ebee80,26193d12db82ca53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:24.243 INFO 4486 --- [ocp-schedules-30,d8f95e59948eee15,59801dfb6db57c96] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:35:24.416 INFO 4486 --- [ocp-schedules-6,29ecf3373092a54d,aa2a9917ae7c1f4c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:35:24.597 INFO 4486 --- [ocp-schedules-5,15b3046de5a08a46,5a183b525747b2b2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:35:24.597 INFO 4486 --- [ocp-schedules-5,15b3046de5a08a46,5a183b525747b2b2] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:35:24.633 INFO 4486 --- [Collect all tenants major compactions2,15b3046de5a08a46,4f3463f81df8399d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:35:24.633 INFO 4486 --- [Collect all tenants major compactions3,15b3046de5a08a46,7f1c8348a09a3424] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:35:24.633 INFO 4486 --- [Collect all tenants major compactions1,15b3046de5a08a46,b9f5b7904afeac19] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:35:24.634 INFO 4486 --- [Collect all tenants major compactions5,15b3046de5a08a46,0d7d47bf68e907f8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:35:24.634 INFO 4486 --- [Collect all tenants major compactions4,15b3046de5a08a46,73232d60093767a0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:35:24.657 INFO 4486 --- [Collect all tenants major compactions1,15b3046de5a08a46,b9f5b7904afeac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:24.658 INFO 4486 --- [Collect all tenants major compactions1,15b3046de5a08a46,b9f5b7904afeac19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:35:24.660 INFO 4486 --- [Collect all tenants major compactions2,15b3046de5a08a46,4f3463f81df8399d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:24.660 INFO 4486 --- [Collect all tenants major compactions3,15b3046de5a08a46,7f1c8348a09a3424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:24.662 INFO 4486 --- [Collect all tenants major compactions2,15b3046de5a08a46,4f3463f81df8399d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:35:24.662 INFO 4486 --- [Collect all tenants major compactions1,15b3046de5a08a46,b9f5b7904afeac19] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:35:24.662 INFO 4486 --- [Collect all tenants major compactions3,15b3046de5a08a46,7f1c8348a09a3424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:35:24.665 INFO 4486 --- [Collect all tenants major compactions3,15b3046de5a08a46,7f1c8348a09a3424] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:35:24.668 INFO 4486 --- [Collect all tenants major compactions1,15b3046de5a08a46,b9f5b7904afeac19] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:35:24.669 INFO 4486 --- [Collect all tenants major compactions2,15b3046de5a08a46,4f3463f81df8399d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:35:24.669 INFO 4486 --- [Collect all tenants major compactions2,15b3046de5a08a46,4f3463f81df8399d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:35:24.670 INFO 4486 --- [Collect all tenants major compactions5,15b3046de5a08a46,0d7d47bf68e907f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:24.671 INFO 4486 --- [Collect all tenants major compactions4,15b3046de5a08a46,73232d60093767a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:24.672 INFO 4486 --- [Collect all tenants major compactions3,15b3046de5a08a46,7f1c8348a09a3424] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:35:24.672 INFO 4486 --- [Collect all tenants major compactions5,15b3046de5a08a46,0d7d47bf68e907f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:35:24.673 INFO 4486 --- [Collect all tenants major compactions4,15b3046de5a08a46,73232d60093767a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:35:24.676 INFO 4486 --- [Collect all tenants major compactions4,15b3046de5a08a46,73232d60093767a0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:35:24.676 INFO 4486 --- [Collect all tenants major compactions4,15b3046de5a08a46,73232d60093767a0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:35:24.677 INFO 4486 --- [Collect all tenants major compactions5,15b3046de5a08a46,0d7d47bf68e907f8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:35:24.677 INFO 4486 --- [Collect all tenants major compactions5,15b3046de5a08a46,0d7d47bf68e907f8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:35:24.678 INFO 4486 --- [ocp-schedules-5,15b3046de5a08a46,5a183b525747b2b2] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:35:24.772 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:25.076 INFO 4486 --- [ocp-async-9,7781fe6bcba61a9e,415298c722aa82ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:25.994 INFO 4486 --- [prometheus-minute-schedule-1,2193ce065598c61e,9a609a20d60a1538] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:26.000 INFO 4486 --- [prometheus-minute-schedule-1,2193ce065598c61e,9a609a20d60a1538] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:26.935 INFO 4486 --- [ocp-schedules-30,46c22516d45ce150,ed2bd392945674de] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:35:26.935+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:35:26.937 INFO 4486 --- [ocp-schedules-30,46c22516d45ce150,ed2bd392945674de] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:35:26.938 INFO 4486 --- [ocp-schedules-30,46c22516d45ce150,ed2bd392945674de] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:35:26.938 INFO 4486 --- [ocp-schedules-30,46c22516d45ce150,ed2bd392945674de] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:35:27.076 INFO 4486 --- [ocp-async-10,1d51a14c29bd4609,b2d6163891d084ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:27.184 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,20c992c7e1c553f8,3be71cfbc2b4496b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=20c992c7e1c553f8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:27.185 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,20c992c7e1c553f8,3be71cfbc2b4496b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=20c992c7e1c553f8, duration=3 ms] 2024-10-16 13:35:27.773 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:28.630 INFO 4486 --- [ocp-schedules-8,3bedb09b1f5792ad,8412cf7d0a210f72] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:35:28.632 INFO 4486 --- [ocp-schedules-8,3bedb09b1f5792ad,8412cf7d0a210f72] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:28.994 INFO 4486 --- [prometheus-minute-schedule-1,c5ff32a7d6a378c3,1ca908dc6cace3d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:28.999 INFO 4486 --- [prometheus-minute-schedule-1,c5ff32a7d6a378c3,1ca908dc6cace3d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:29.076 INFO 4486 --- [ocp-async-4,73b60bfae27ba8a5,5604608f60d1b284] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:29.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,35d0176049734b2d,bd289caf4afae7d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=35d0176049734b2d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:29.348 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,35d0176049734b2d,bd289caf4afae7d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=35d0176049734b2d, duration=1 ms] 2024-10-16 13:35:30.103 INFO 4486 --- [prometheus-second-schedule-1,8d9d4ce533cb9fd3,e41285cee592e098] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:35:30.365 INFO 4486 --- [ocp-schedules-9,c80c4d3566d1f48e,5e06367fbb853c46] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:35:30.775 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:30.935 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d1e847886e22856b,2f2c54ab170b2695] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d1e847886e22856b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:30.935 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d1e847886e22856b,2f2c54ab170b2695] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d1e847886e22856b, duration=1 ms] 2024-10-16 13:35:31.076 INFO 4486 --- [ocp-async-5,0095238bb6fdac4c,67505b933e6c4650] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:31.285 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b8b2a6f56c6d2744,5e882b6d214af327] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b8b2a6f56c6d2744, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:31.285 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b8b2a6f56c6d2744,5e882b6d214af327] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b8b2a6f56c6d2744, duration=1 ms] 2024-10-16 13:35:31.758 INFO 4486 --- [ocp-schedules-13,58afdbb1c50e1ad1,1fb364295802e3b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:35:31.787 WARN 4486 --- [scanner-13254337-1,58afdbb1c50e1ad1,6a03bf12e10f8f9b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:35:31.859 INFO 4486 --- [alarm-detect-4,58afdbb1c50e1ad1,516f877c2400a47a] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056900 ~ 1729056921 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056915, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056915, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056915=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:35:31.965 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8ce443ddad2e0234,b2f8b0158b678f9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=8ce443ddad2e0234, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:35:31.965 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8ce443ddad2e0234,b2f8b0158b678f9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=8ce443ddad2e0234, duration=1 ms] 2024-10-16 13:35:31.994 INFO 4486 --- [prometheus-minute-schedule-1,61441cc407c75644,a752485353b9bebd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:32.000 INFO 4486 --- [prometheus-minute-schedule-1,61441cc407c75644,a752485353b9bebd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:32.990 INFO 4486 --- [ocp-async-9,4b46ab633b3016f9,ee717288722bd894] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:35:33.076 INFO 4486 --- [ocp-async-3,a2eb0ac428629ab5,abad1bb93d888d05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:33.225 INFO 4486 --- [ocp-schedules-16,833437b29be7ad37,9d911577d9ac3089] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:35:33.225 INFO 4486 --- [ocp-schedules-16,34ec3f771a04fb91,99c2bef075299c57] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:35:33.227 INFO 4486 --- [ocp-schedules-16,34ec3f771a04fb91,99c2bef075299c57] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:33.252 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.253 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:33.263 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.264 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:33.284 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.285 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.285 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:35:33.288 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.289 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:33.314 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.315 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.316 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:33.317 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:35:33.317 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.320 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:35:33.321 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:33.331 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.331 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:33.342 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.351 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.351 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.352 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:33.353 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:35:33.358 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:35:33.362 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:33.364 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:33.391 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:35:33.396 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.397 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.399 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:35:33.457 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:35:33.457 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:35:33.460 INFO 4486 --- [ocp-schedules-3,4c5d1b7a9bbf8ecb,d4f16e6d852c8bd1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:35:33.461 INFO 4486 --- [ocp-schedules-3,4c5d1b7a9bbf8ecb,d4f16e6d852c8bd1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:33.461 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:35:33.461 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:35:33.462 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:35:33.478 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.480 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.480 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:35:33.483 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:35:33.484 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.488 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:35:33.491 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:33.494 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.495 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.496 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:35:33.507 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:35:33.507 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:35:33.519 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.519 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:33.545 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.545 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:33.567 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.567 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:33.591 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.591 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:35:33.616 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.616 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:35:33.640 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:35:33.665 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.667 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.668 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:33.672 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:35:33.676 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:35:33.681 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:35:33.682 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:35:33.707 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:35:33.775 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:33.843 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:35:33.873 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:33.875 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:33.876 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:35:33.909 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:35:33.909 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:35:34.994 INFO 4486 --- [prometheus-minute-schedule-1,10ac6d613ef71f42,8be26a7ba633db0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:35.000 INFO 4486 --- [prometheus-minute-schedule-1,10ac6d613ef71f42,8be26a7ba633db0f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:35.077 INFO 4486 --- [ocp-async-2,eba05c6a05008789,4034adcdb6eb21c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:36.776 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:37.063 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:35:37.064 INFO 4486 --- [ocp-schedules-18,1f905703857e7b7a,b45b560b7aacd1f4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:35:37.065 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:35:37.066 INFO 4486 --- [ocp-schedules-17,dd665193ec0d5798,47c964d2e34b00c7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:35:37.068 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:35:37.069 INFO 4486 --- [ocp-schedules-17,dd665193ec0d5798,47c964d2e34b00c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:37.071 INFO 4486 --- [ocp-schedules-18,1f905703857e7b7a,b45b560b7aacd1f4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:35:37.077 INFO 4486 --- [ocp-async-5,23b1e0c52921d403,73695c16de81a98e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:37.081 INFO 4486 --- [ocp-schedules-24,55161af97edadd70,770297a1cba7a318] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:35:37.083 INFO 4486 --- [ocp-schedules-24,55161af97edadd70,770297a1cba7a318] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:35:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:35:37.089 INFO 4486 --- [ocp-schedules-26,8d001053f6ef48c2,be31d266e1808e5b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:35:37.089 INFO 4486 --- [ocp-schedules-19,20c147f4fc19ec5f,9bc78fac5c5aa78c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:35:37.090 INFO 4486 --- [host-check-agent-status1,8d001053f6ef48c2,f1c9e9ed34d4d31d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:35:37.090 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:35:37.090 INFO 4486 --- [host-check-agent-status2,8d001053f6ef48c2,3eaf7508a38bef60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:35:37.090 INFO 4486 --- [ocp-schedules-5,4192bbc1e4c401d3,9884d5ddb35fdc1a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:35:37.090 INFO 4486 --- [host-check-agent-status3,8d001053f6ef48c2,40479668d597672f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:35:37.091 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:35:37.091 INFO 4486 --- [host-check-agent-status4,8d001053f6ef48c2,b7d67f0b8953059d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:35:37.091 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:35:37.091 INFO 4486 --- [host-check-clockdiff1,4192bbc1e4c401d3,5b5b702da2886c92] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:35:37.091 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:35:37.092 INFO 4486 --- [host-check-clockdiff2,4192bbc1e4c401d3,9776af3a5ff539df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:35:37.093 INFO 4486 --- [host-check-clockdiff3,4192bbc1e4c401d3,00621670cb4e00ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:35:37.093 INFO 4486 --- [host-check-clockdiff4,4192bbc1e4c401d3,01965961a0b94b1c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:35:37.097 INFO 4486 --- [host-check-agent-status3,8d001053f6ef48c2,40479668d597672f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:35:37.097 INFO 4486 --- [host-check-agent-status1,8d001053f6ef48c2,f1c9e9ed34d4d31d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:37.100 INFO 4486 --- [host-check-agent-status4,8d001053f6ef48c2,b7d67f0b8953059d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:35:37.100 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:37.101 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:35:37.102 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:35:37.103 INFO 4486 --- [host-check-agent-status4,8d001053f6ef48c2,b7d67f0b8953059d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:35:37.103 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:37.103 INFO 4486 --- [host-check-agent-status1,8d001053f6ef48c2,f1c9e9ed34d4d31d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:37.121 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:35:37.103 INFO 4486 --- [host-check-agent-status3,8d001053f6ef48c2,40479668d597672f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:35:37.121 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:35:37.131 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:35:37.131 INFO 4486 --- [host-check-agent-status2,8d001053f6ef48c2,3eaf7508a38bef60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:35:37.131 INFO 4486 --- [host-check-clockdiff1,4192bbc1e4c401d3,5b5b702da2886c92] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056937 0 0 , error= 2024-10-16 13:35:37.131 INFO 4486 --- [host-check-clockdiff4,4192bbc1e4c401d3,01965961a0b94b1c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056937 3 3 , error= 2024-10-16 13:35:37.135 INFO 4486 --- [host-check-clockdiff2,4192bbc1e4c401d3,9776af3a5ff539df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056937 3 3 , error= 2024-10-16 13:35:37.138 INFO 4486 --- [host-check-agent-status2,8d001053f6ef48c2,3eaf7508a38bef60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:35:37.138 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:35:37.138 INFO 4486 --- [host-check-clockdiff3,4192bbc1e4c401d3,00621670cb4e00ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056937 3 3 , error= 2024-10-16 13:35:37.147 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.148 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:37.150 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:35:37.161 INFO 4486 --- [host-check-agent-status3,8d001053f6ef48c2,40479668d597672f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:35:37.163 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:35:37.163 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.163 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:35:37.164 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:35:37.165 INFO 4486 --- [host-check-agent-status1,8d001053f6ef48c2,f1c9e9ed34d4d31d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:35:37.166 INFO 4486 --- [host-check-agent-status2,8d001053f6ef48c2,3eaf7508a38bef60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:35:37.166 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:35:37.168814631+08:00 2024-10-16 13:35:37.166 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:35:37.165152243+08:00 2024-10-16 13:35:37.166 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:35:37.167 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:35:37.166+08:00 2024-10-16 13:35:37.167 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:37.167 INFO 4486 --- [host-check-agent-status3,8d001053f6ef48c2,40479668d597672f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:35:37.167 INFO 4486 --- [host-check-agent-status3,8d001053f6ef48c2,40479668d597672f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:35:37.167 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:35:37.166+08:00 2024-10-16 13:35:37.167 INFO 4486 --- [host-check-agent-status3,8d001053f6ef48c2,40479668d597672f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:35:37.168 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:35:37.168 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:35:37.171165385+08:00 2024-10-16 13:35:37.169 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-agent-status2,8d001053f6ef48c2,3eaf7508a38bef60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-agent-status2,8d001053f6ef48c2,3eaf7508a38bef60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:35:37.168+08:00 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-agent-status2,8d001053f6ef48c2,3eaf7508a38bef60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-agent-status1,8d001053f6ef48c2,f1c9e9ed34d4d31d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-agent-status1,8d001053f6ef48c2,f1c9e9ed34d4d31d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-agent-status1,8d001053f6ef48c2,f1c9e9ed34d4d31d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:35:37.166+08:00 and status AVAILABLE 2024-10-16 13:35:37.176 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:35:37.166+08:00 and status AVAILABLE 2024-10-16 13:35:37.177 INFO 4486 --- [host-check-agent-status4,8d001053f6ef48c2,b7d67f0b8953059d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:35:37.178 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:35:37.181278868+08:00 2024-10-16 13:35:37.179 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:35:37.168+08:00 and status AVAILABLE 2024-10-16 13:35:37.179 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:35:37.179 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:35:37.178+08:00 2024-10-16 13:35:37.179 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:35:37.180 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.181 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:35:37.181 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:35:37.182 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:35:37.178+08:00 and status AVAILABLE 2024-10-16 13:35:37.182 INFO 4486 --- [host-check-agent-status4,8d001053f6ef48c2,b7d67f0b8953059d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:35:37.182 INFO 4486 --- [host-check-agent-status4,8d001053f6ef48c2,b7d67f0b8953059d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:35:37.182 INFO 4486 --- [host-check-agent-status4,8d001053f6ef48c2,b7d67f0b8953059d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:35:37.182 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.183 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.183 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.185 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:35:37.185 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:35:37.187 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:35:37.188 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:35:37.188 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:37.189 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:35:37.190 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:37.190 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:35:37.191 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2a5dc7d580ab2539,6caac31db020619d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2a5dc7d580ab2539, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:37.191 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2a5dc7d580ab2539,6caac31db020619d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2a5dc7d580ab2539, duration=1 ms] 2024-10-16 13:35:37.191 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:35:37.194 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.195 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:35:37.195 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:35:37.196 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.196 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.196 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.196 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.196 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.196 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.197 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:35:37.205 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.207 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:35:37.213 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:35:37.213 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:37.218 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:37.219 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:37.221 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.222 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:35:37.223 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.225 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:35:37.227 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:35:37.234 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.235 INFO 4486 --- [ocp-schedules-9,d0a42d7299fff93e,8dc6497928702df1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:35:37.247 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:35:37.256 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:35:37.257 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:35:37.266 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:35:37.267 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:35:37.271 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:37.273 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:35:37.277 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:35:37.283 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:35:37.285 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:37.295 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:37.300 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:35:37.300 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:35:37.309 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:35:37.317 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:35:37.324 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:37.328 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:35:37.333 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:35:37.343 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:35:37.345 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:37.354 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:35:37.359 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:37.365 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:35:37.367 INFO 4486 --- [host-check-host-status3,20c147f4fc19ec5f,7dcdaa91ebb8a9c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:35:37.377 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:35:37.383 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:35:37.394 INFO 4486 --- [host-check-host-status4,20c147f4fc19ec5f,f0d997bd1acb6c5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:35:37.394 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:35:37.398 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:35:37.407 INFO 4486 --- [host-check-host-status2,20c147f4fc19ec5f,9b077ab03120ea65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:35:37.431 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:37.433 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.434 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:35:37.435 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.436 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.437 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:35:37.446 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:35:37.448 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.448 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.448 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.448 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.448 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.449 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.457 INFO 4486 --- [host-check-host-status1,20c147f4fc19ec5f,cc3f2d4ec94b7793] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:35:37.477 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:35:37.520 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:35:37.569 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:37.584 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:37.586 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.586 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:35:37.589 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:35:37.594 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.595 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:35:37.596 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.596 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.597 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:35:37.609 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.609 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.609 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.609 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.609 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.610 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.617 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:35:37.810 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:35:37.856 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:37.860 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.863 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:35:37.864 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.865 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.865 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:35:37.876 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.876 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.876 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.876 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.876 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:37.876 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:37.894 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:35:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:35:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:35:37.994 INFO 4486 --- [prometheus-minute-schedule-1,811d7259a72882ff,e40a3132610c4d11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:38.000 INFO 4486 --- [prometheus-minute-schedule-1,811d7259a72882ff,e40a3132610c4d11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:38.094 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:35:38.141 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:35:38.145 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:38.146 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:35:38.149 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:38.149 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:38.150 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:35:38.160 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:38.160 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:38.160 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:38.160 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:38.160 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:35:38.161 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:38.225 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:35:38.436 INFO 4486 --- [ocp-schedules-5,22e6cf7c108ae2e3,ff1605271fe203da] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:35:38.437 INFO 4486 --- [ocp-schedules-5,22e6cf7c108ae2e3,ff1605271fe203da] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:35:38.437 INFO 4486 --- [ocp-schedules-5,22e6cf7c108ae2e3,ff1605271fe203da] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:35:38.458 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:35:38.228974, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:35:38.458 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:35:38.228974, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:35:38.467 INFO 4486 --- [ocp-schedules-9,23e5793aaf40b068,518404d313e9e67b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:35:38.467 INFO 4486 --- [ocp-schedules-9,23e5793aaf40b068,518404d313e9e67b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:35:38.499 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:35:38.228974} } 2024-10-16 13:35:38.564 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:35:38.573 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:35:38.576 INFO 4486 --- [ocp-schedules-23,28237ff33b4bf9f0,04be9143cab78fd2] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:35:38.662 INFO 4486 --- [ocp-schedules-8,3bedb09b1f5792ad,8412cf7d0a210f72] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:35:38.889 INFO 4486 --- [ocp-schedules-2,46d0001561f27fd0,eb5b0033ea4b01ec] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:35:39.077 INFO 4486 --- [ocp-async-10,d16ca001ba0c5d31,6d908df488ce496f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:39.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,295555c6106af437,21224ea22a1495ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=295555c6106af437, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:39.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,295555c6106af437,21224ea22a1495ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=295555c6106af437, duration=2 ms] 2024-10-16 13:35:39.777 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:40.006 INFO 4486 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:35:40.007 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:35:40.027 INFO 4486 --- [Check obproxy1,c04bca2b6025e1b0,0020b3d2f43bea60] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:35:40.028 INFO 4486 --- [Check obproxy2,06bc3a5e2f2a4821,5c07ad25a60b9299] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:35:40.028 INFO 4486 --- [Check obproxy2,06bc3a5e2f2a4821,5c07ad25a60b9299] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:40.028 INFO 4486 --- [Check obproxy1,c04bca2b6025e1b0,0020b3d2f43bea60] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:40.031 INFO 4486 --- [Collect obproxy2,e6c3ae9fff526454,e36bf00f18fd8f08] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:35:40.031 INFO 4486 --- [Collect obproxy2,e6c3ae9fff526454,e36bf00f18fd8f08] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:40.032 INFO 4486 --- [Collect obproxy1,79a5fcbc78034b78,cac1815b4283a207] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:35:40.032 INFO 4486 --- [Collect obproxy1,79a5fcbc78034b78,cac1815b4283a207] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:35:40.051 INFO 4486 --- [Check obproxy2,06bc3a5e2f2a4821,5c07ad25a60b9299] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:35:40.051 INFO 4486 --- [Check obproxy2,06bc3a5e2f2a4821,5c07ad25a60b9299] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:35:40.052 INFO 4486 --- [Collect obproxy2,e6c3ae9fff526454,e36bf00f18fd8f08] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:35:40.053 INFO 4486 --- [Check obproxy1,c04bca2b6025e1b0,0020b3d2f43bea60] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:35:40.053 INFO 4486 --- [Check obproxy1,c04bca2b6025e1b0,0020b3d2f43bea60] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:35:40.053 INFO 4486 --- [Collect obproxy1,79a5fcbc78034b78,cac1815b4283a207] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:35:40.057 INFO 4486 --- [Collect obproxy2,e6c3ae9fff526454,e36bf00f18fd8f08] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:35:40.058 INFO 4486 --- [Collect obproxy1,79a5fcbc78034b78,cac1815b4283a207] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:35:40.060 INFO 4486 --- [Collect obproxy1,79a5fcbc78034b78,cac1815b4283a207] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:35:40.061 INFO 4486 --- [Collect obproxy2,e6c3ae9fff526454,e36bf00f18fd8f08] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:35:40.370 INFO 4486 --- [ocp-schedules-10,233b19746fa6bbcf,60c43ce3588641bf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:35:40.947 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,55bc853ce869fcd4,82e0ac9f2357e8b3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=55bc853ce869fcd4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:40.947 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,55bc853ce869fcd4,82e0ac9f2357e8b3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=55bc853ce869fcd4, duration=2 ms] 2024-10-16 13:35:40.994 INFO 4486 --- [prometheus-minute-schedule-1,f1e6cb5b5f142444,d348aeb6b1d881b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:40.995 INFO 4486 --- [prometheus-hour-schedule-1,9ffd0d9654607a23,81532e2676e43800] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:35:41.001 INFO 4486 --- [prometheus-minute-schedule-1,f1e6cb5b5f142444,d348aeb6b1d881b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:41.077 INFO 4486 --- [ocp-async-2,65f1c7d09ed1a10a,4008ccec7be9e383] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:41.294 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2de80b39f23c9045,de19af0ac82cdb8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2de80b39f23c9045, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:41.294 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2de80b39f23c9045,de19af0ac82cdb8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2de80b39f23c9045, duration=2 ms] 2024-10-16 13:35:41.943 INFO 4486 --- [ocp-schedules-8,6926fc27cf0063b3,ad467d888cc1f42f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:35:41.973 WARN 4486 --- [scanner-13254690-1,6926fc27cf0063b3,3c48df18d02c916c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:35:42.030 INFO 4486 --- [alarm-detect-17,6926fc27cf0063b3,f3ec22314a5cbb2b] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056915 ~ 1729056931 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056930, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056930, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056930=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:35:42.089 INFO 4486 --- [ocp-schedules-4,2ea50651a60dee6b,0fd9403ccdf342ae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:35:42.778 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:42.990 INFO 4486 --- [ocp-async-7,2af63ae4056891b2,fde955612262e312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:35:43.077 INFO 4486 --- [ocp-async-5,225bf1fb5cecf071,9b8124f22ce7be34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:43.462 INFO 4486 --- [ocp-schedules-3,4c5d1b7a9bbf8ecb,d4f16e6d852c8bd1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:35:43.994 INFO 4486 --- [prometheus-minute-schedule-1,4034b31b83c92523,cdb3fe17dc58a14a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:44.000 INFO 4486 --- [prometheus-minute-schedule-1,4034b31b83c92523,cdb3fe17dc58a14a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:45.078 INFO 4486 --- [ocp-async-10,ed5c0b1ab6d65823,836f8fabe4485e5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:45.454 INFO 4486 --- [ocp-schedules-26,0f348b6d16757714,8a1d1a70a563586d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:35:45.459 INFO 4486 --- [ocp-schedules-26,0f348b6d16757714,8a1d1a70a563586d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:35:45.459 INFO 4486 --- [ocp-schedules-26,0f348b6d16757714,8a1d1a70a563586d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:35:45.475 INFO 4486 --- [ocp-schedules-20,05d5e9b355af3ade,5b630c34facec83c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:35:45.475 INFO 4486 --- [ocp-schedules-20,05d5e9b355af3ade,5b630c34facec83c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:35:45.779 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:46.994 INFO 4486 --- [prometheus-minute-schedule-1,ce37e43538ff83c1,9541c7f5415ccec5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:47.000 INFO 4486 --- [prometheus-minute-schedule-1,ce37e43538ff83c1,9541c7f5415ccec5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:47.070 INFO 4486 --- [ocp-schedules-17,dd665193ec0d5798,47c964d2e34b00c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:35:47.078 INFO 4486 --- [ocp-async-2,e3572ffb9b9a29a5,3f71d2350f6fd855] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:47.102 INFO 4486 --- [ocp-schedules-25,e4974cfd02bbdd68,86a94263ea3bff45] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:35:47.168 INFO 4486 --- [ocp-schedules-24,55161af97edadd70,770297a1cba7a318] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:35:47.196 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,085ee8ef69b0e63d,23d87c35043dd6ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=085ee8ef69b0e63d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:47.196 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,085ee8ef69b0e63d,23d87c35043dd6ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=085ee8ef69b0e63d, duration=1 ms] 2024-10-16 13:35:48.780 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:48.781 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:48.781 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:35:48.782 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:35:48.783 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:35:48.784 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:35:49.078 INFO 4486 --- [ocp-async-9,a78c614df8225bdc,542692ca6998c854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:49.356 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0261173faf69c9cc,b25a4dffba05f312] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0261173faf69c9cc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:49.356 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,0261173faf69c9cc,b25a4dffba05f312] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0261173faf69c9cc, duration=1 ms] 2024-10-16 13:35:49.994 INFO 4486 --- [prometheus-minute-schedule-1,99a004662ea0d74e,ee6bf93ea5226c9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:50.001 INFO 4486 --- [prometheus-minute-schedule-1,99a004662ea0d74e,ee6bf93ea5226c9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:50.090 INFO 4486 --- [ocp-schedules-25,6442a09d9ebad90a,10300bce2676ef82] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:35:50.090 INFO 4486 --- [ocp-schedules-25,6442a09d9ebad90a,10300bce2676ef82] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:35:50.090 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:35:50.090 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:35:50.090 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:35:50.090 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:35:50.090 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:35:50.096 INFO 4486 --- [ocp-schedules-25,6442a09d9ebad90a,10300bce2676ef82] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:35:50.097 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:35:50.097 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:35:50.097 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:35:50.097 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:35:50.098 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:35:50.098 INFO 4486 --- [ocp-schedules-25,6442a09d9ebad90a,10300bce2676ef82] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:35:50.098 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:35:50.098 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:35:50.098 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:35:50.099 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:35:50.099 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:35:50.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:35:50.123 INFO 4486 --- [ocp-schedules-25,6442a09d9ebad90a,10300bce2676ef82] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:35:50.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:35:50.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:35:50.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:35:50.125 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:35:50.130 INFO 4486 --- [ocp-schedules-25,6442a09d9ebad90a,10300bce2676ef82] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:35:50.376 INFO 4486 --- [ocp-schedules-10,5b285ec805df1632,acba63d65823f07a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:35:50.380 INFO 4486 --- [ocp-schedules-29,4658062e5b598e9b,2431e59709534b7e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:35:50.380 INFO 4486 --- [ocp-schedules-29,4658062e5b598e9b,2431e59709534b7e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:35:50.386 INFO 4486 --- [ocp-schedules-29,4658062e5b598e9b,2431e59709534b7e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:35:50.386 INFO 4486 --- [ocp-schedules-29,4658062e5b598e9b,2431e59709534b7e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:35:50.389 INFO 4486 --- [ocp-schedules-29,4658062e5b598e9b,2431e59709534b7e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:35:50.389 INFO 4486 --- [ocp-schedules-29,4658062e5b598e9b,2431e59709534b7e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:35:50.951 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8bf7509891ca9ff3,06281b8a4fa6f97e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8bf7509891ca9ff3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:50.951 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8bf7509891ca9ff3,06281b8a4fa6f97e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8bf7509891ca9ff3, duration=0 ms] 2024-10-16 13:35:51.078 INFO 4486 --- [ocp-async-3,1383e5707fdeb6c2,280f6d9883bc60d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:51.298 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ccdddcaf44f12453,0cd7b4d7b0109a91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ccdddcaf44f12453, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:51.298 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ccdddcaf44f12453,0cd7b4d7b0109a91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ccdddcaf44f12453, duration=1 ms] 2024-10-16 13:35:51.780 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:52.074 INFO 4486 --- [ocp-schedules-28,9d0673c995e0b2ed,f72680fb6d21cc08] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:35:52.087 WARN 4486 --- [scanner-13254917-1,9d0673c995e0b2ed,3958140a26e56221] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:35:52.162 INFO 4486 --- [ocp-schedules-10,05e33b41afc61a4f,7d7375466f37a91d] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:35:52.991 INFO 4486 --- [ocp-async-6,6b0912bce3401269,aaba0504d45dd7f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:35:52.994 INFO 4486 --- [prometheus-minute-schedule-1,935c424dfc256452,b304f97b09bac544] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:53.000 INFO 4486 --- [prometheus-minute-schedule-1,935c424dfc256452,b304f97b09bac544] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:53.078 INFO 4486 --- [ocp-async-7,2fabe7aa706f47ff,20957daa6a46a9c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:54.781 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:55.079 INFO 4486 --- [ocp-async-5,9a2d3d9c18d61704,9abd1c943229b39e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:55.994 INFO 4486 --- [prometheus-minute-schedule-1,f2e228d519d5851a,4d82159c6be5c7b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:56.002 INFO 4486 --- [prometheus-minute-schedule-1,f2e228d519d5851a,4d82159c6be5c7b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:56.864 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,02bb4fc0a0b56200,74c79589c3aa2abe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=02bb4fc0a0b56200, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:35:56.865 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,02bb4fc0a0b56200,74c79589c3aa2abe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=02bb4fc0a0b56200, duration=3 ms] 2024-10-16 13:35:57.079 INFO 4486 --- [ocp-async-10,3f5730d297640dde,b8fe8ecbcb44dd85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:57.200 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ba89d504f1bb10f7,3f96a0260839685b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ba89d504f1bb10f7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:57.200 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ba89d504f1bb10f7,3f96a0260839685b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ba89d504f1bb10f7, duration=1 ms] 2024-10-16 13:35:57.782 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:35:58.893 INFO 4486 --- [ocp-schedules-20,22399385a9dbb5df,cad45caaf0293d34] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:35:58.994 INFO 4486 --- [prometheus-minute-schedule-1,4a1819b4f9356332,30bfdbc3dc1c27c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729056960, interval=60 2024-10-16 13:35:59.001 INFO 4486 --- [prometheus-minute-schedule-1,4a1819b4f9356332,30bfdbc3dc1c27c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:35:59.079 INFO 4486 --- [ocp-async-4,3c045e23e9389e98,b6d64eef2e91ac2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:35:59.417 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a551f223e2ec5df6,c0c8bbb11aabbf18] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a551f223e2ec5df6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:35:59.417 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a551f223e2ec5df6,c0c8bbb11aabbf18] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a551f223e2ec5df6, duration=1 ms] 2024-10-16 13:36:00.383 INFO 4486 --- [ocp-schedules-29,32fb95729b1b3d4a,1d36a87707a1949e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:36:00.783 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:00.955 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,98b73ae1934d37db,82b2d0b1746fe215] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=98b73ae1934d37db, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:00.956 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,98b73ae1934d37db,82b2d0b1746fe215] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=98b73ae1934d37db, duration=1 ms] 2024-10-16 13:36:01.079 INFO 4486 --- [ocp-async-7,94c11d252a6d4b23,df4e7e38f7dd4c1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:01.219 INFO 4486 --- [ocp-schedules-21,95920c878ca303d5,848c0644f904793a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:36:01.219+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:36:01.224 INFO 4486 --- [ocp-schedules-21,95920c878ca303d5,848c0644f904793a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:36:01.228 INFO 4486 --- [ocp-schedules-21,95920c878ca303d5,848c0644f904793a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:36:01.238 INFO 4486 --- [ocp-schedules-21,95920c878ca303d5,848c0644f904793a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:36:01.302 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7bcc5b355ffb5acc,87d88af023c753c5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7bcc5b355ffb5acc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:01.303 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7bcc5b355ffb5acc,87d88af023c753c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7bcc5b355ffb5acc, duration=2 ms] 2024-10-16 13:36:01.994 INFO 4486 --- [prometheus-minute-schedule-1,ba4f05a2ca2ca1ff,ee868215b85bd118] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:02.001 INFO 4486 --- [prometheus-minute-schedule-1,ba4f05a2ca2ca1ff,ee868215b85bd118] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:02.251 INFO 4486 --- [ocp-schedules-10,407398ce45bdc0a2,ad9b6b57f8834ce6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:36:02.282 WARN 4486 --- [scanner-13255257-1,407398ce45bdc0a2,0f2d4b0bd1f7467a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:36:02.310 INFO 4486 --- [alarm-detect-1,407398ce45bdc0a2,bf73417c25f783bd] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056930 ~ 1729056952 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056945, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056945, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056945=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:36:02.991 INFO 4486 --- [ocp-async-9,e81101e382395378,5fe449eb48edffa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:36:03.079 INFO 4486 --- [ocp-async-3,efcffe8d6ef104f2,b347af13b2ef5836] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:03.784 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:03.911 INFO 4486 --- [ocp-schedules-16,34ec3f771a04fb91,99c2bef075299c57] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:36:03.911 INFO 4486 --- [ocp-schedules-16,b461b1e5570bfb69,9d63003d89cbca01] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:36:03.913 INFO 4486 --- [ocp-schedules-16,b461b1e5570bfb69,9d63003d89cbca01] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:03.941 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:03.943 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:03.958 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:03.959 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:03.980 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:03.981 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:03.982 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:36:03.984 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:03.985 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:04.012 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:04.013 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.013 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:04.015 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:36:04.015 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.018 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:36:04.020 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:04.022 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a220ca7fcc710594,b7f4842f9c3fe962] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=a220ca7fcc710594, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:36:04.022 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a220ca7fcc710594,b7f4842f9c3fe962] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=a220ca7fcc710594, duration=1 ms] 2024-10-16 13:36:04.030 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.030 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:04.043 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.052 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:04.052 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.053 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:04.054 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:36:04.059 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:36:04.064 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:04.066 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:04.089 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:36:04.097 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:04.099 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.099 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:36:04.131 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:36:04.131 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:36:04.148 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:04.150 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.150 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:04.155 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.159 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:36:04.161 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:36:04.162 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:04.170 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:04.171 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.172 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:36:04.180 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:36:04.180 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:36:04.186 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.186 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:04.213 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.213 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:04.236 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.236 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:04.265 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.265 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:04.290 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.290 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:04.316 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:04.342 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:04.343 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.344 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:04.349 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:36:04.354 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:36:04.358 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:36:04.360 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:36:04.383 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:36:04.519 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:36:04.547 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:04.549 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:04.550 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:36:04.580 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:36:04.580 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:36:04.994 INFO 4486 --- [prometheus-minute-schedule-1,4d1c8672b2221de5,e054837e3570e9dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:05.001 INFO 4486 --- [prometheus-minute-schedule-1,4d1c8672b2221de5,e054837e3570e9dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:05.080 INFO 4486 --- [ocp-async-2,7f1de9d999b84e79,9945a18df2b87252] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:06.785 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:07.066 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:36:07.067 INFO 4486 --- [ocp-schedules-20,1c1e5f226a0ac0d9,762eb6ff721b3e08] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:36:07.068 INFO 4486 --- [ocp-schedules-20,1c1e5f226a0ac0d9,762eb6ff721b3e08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:07.080 INFO 4486 --- [ocp-async-7,4c8276e9446abf12,22ce4f85cb5cbe09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:07.081 INFO 4486 --- [ocp-schedules-21,796dd645470eec7e,c595a69a49fbd7d1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:36:07.083 INFO 4486 --- [ocp-schedules-21,796dd645470eec7e,c595a69a49fbd7d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:36:07.088 INFO 4486 --- [ocp-schedules-25,28a6d0a2b8fb90e9,95529f0084a96f43] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:36:07.088 INFO 4486 --- [ocp-schedules-11,e310abd214b29d24,d60db51b9fc51d6f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:36:07.089 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:36:07.089 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:36:07.089 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:36:07.089 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:36:07.089 INFO 4486 --- [ocp-schedules-2,2b95f45cbea5c001,6a7d29fd65c8cffe] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:36:07.089 INFO 4486 --- [host-check-agent-status1,e310abd214b29d24,b9a40dd8c01dd418] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:36:07.089 INFO 4486 --- [host-check-agent-status2,e310abd214b29d24,7ad6da5ef923bc8d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:36:07.090 INFO 4486 --- [host-check-agent-status3,e310abd214b29d24,1e9f39080347f475] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:36:07.090 INFO 4486 --- [host-check-clockdiff1,2b95f45cbea5c001,045f48c2ec6b2c9d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:36:07.090 INFO 4486 --- [host-check-agent-status4,e310abd214b29d24,20d6d2df9282d8b3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:36:07.090 INFO 4486 --- [host-check-clockdiff2,2b95f45cbea5c001,408253939197bc1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:36:07.090 INFO 4486 --- [host-check-clockdiff3,2b95f45cbea5c001,dc252b443b1d011b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:36:07.092 INFO 4486 --- [host-check-clockdiff4,2b95f45cbea5c001,b7f8ab40561c7c4e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:36:07.095 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:36:07.096 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:36:07.097 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:36:07.097 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:36:07.097 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:07.098 INFO 4486 --- [host-check-agent-status2,e310abd214b29d24,7ad6da5ef923bc8d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:36:07.098 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:36:07.098 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:36:07.099 INFO 4486 --- [host-check-agent-status3,e310abd214b29d24,1e9f39080347f475] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:36:07.100 INFO 4486 --- [host-check-agent-status1,e310abd214b29d24,b9a40dd8c01dd418] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:07.116 INFO 4486 --- [host-check-agent-status2,e310abd214b29d24,7ad6da5ef923bc8d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:36:07.116 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:07.116 INFO 4486 --- [host-check-agent-status3,e310abd214b29d24,1e9f39080347f475] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:36:07.122 INFO 4486 --- [host-check-clockdiff1,2b95f45cbea5c001,045f48c2ec6b2c9d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056967 0 0 , error= 2024-10-16 13:36:07.123 INFO 4486 --- [host-check-agent-status1,e310abd214b29d24,b9a40dd8c01dd418] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:07.124 INFO 4486 --- [host-check-agent-status4,e310abd214b29d24,20d6d2df9282d8b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:36:07.126 INFO 4486 --- [host-check-agent-status4,e310abd214b29d24,20d6d2df9282d8b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:36:07.134 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.137 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:07.138 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:36:07.179 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:36:07.179 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:36:07.179 INFO 4486 --- [host-check-agent-status2,e310abd214b29d24,7ad6da5ef923bc8d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:36:07.179 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:36:07.184 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:36:07.186186540+08:00 2024-10-16 13:36:07.184 INFO 4486 --- [host-check-agent-status3,e310abd214b29d24,1e9f39080347f475] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:36:07.184 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:36:07.186735962+08:00 2024-10-16 13:36:07.184 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:36:07.184 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:36:07.184+08:00 2024-10-16 13:36:07.184 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:36:07.184+08:00 2024-10-16 13:36:07.185 INFO 4486 --- [host-check-agent-status1,e310abd214b29d24,b9a40dd8c01dd418] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:36:07.185 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:36:07.184783982+08:00 2024-10-16 13:36:07.185 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:36:07.185+08:00 2024-10-16 13:36:07.187 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:36:07.189787173+08:00 2024-10-16 13:36:07.187 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:36:07.187+08:00 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:36:07.184+08:00 and status AVAILABLE 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:36:07.184+08:00 and status AVAILABLE 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status4,e310abd214b29d24,20d6d2df9282d8b3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:36:07.185+08:00 and status AVAILABLE 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status2,e310abd214b29d24,7ad6da5ef923bc8d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status2,e310abd214b29d24,7ad6da5ef923bc8d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status2,e310abd214b29d24,7ad6da5ef923bc8d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status1,e310abd214b29d24,b9a40dd8c01dd418] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status1,e310abd214b29d24,b9a40dd8c01dd418] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status1,e310abd214b29d24,b9a40dd8c01dd418] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:36:07.188 INFO 4486 --- [host-check-agent-status3,e310abd214b29d24,1e9f39080347f475] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:36:07.189 INFO 4486 --- [host-check-agent-status3,e310abd214b29d24,1e9f39080347f475] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:36:07.189 INFO 4486 --- [host-check-agent-status3,e310abd214b29d24,1e9f39080347f475] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:36:07.190 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:36:07.187+08:00 and status AVAILABLE 2024-10-16 13:36:07.190 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:36:07.190 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:36:07.190 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:36:07.192 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:36:07.192 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.192 INFO 4486 --- [host-check-agent-status4,e310abd214b29d24,20d6d2df9282d8b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:36:07.192 INFO 4486 --- [host-check-agent-status4,e310abd214b29d24,20d6d2df9282d8b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:36:07.192 INFO 4486 --- [host-check-agent-status4,e310abd214b29d24,20d6d2df9282d8b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:36:07.193 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:07.193 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.197 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:36:07.198 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:36:07.198 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:36:07.199 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:07.199 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:36:07.200 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:36:07.200 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:36:07.200 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:07.201 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:36:07.203 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:36:07.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b13515a7409bb502,2bfc9fd4c6c274f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b13515a7409bb502, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:07.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b13515a7409bb502,2bfc9fd4c6c274f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b13515a7409bb502, duration=1 ms] 2024-10-16 13:36:07.205 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.206 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:07.217 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.218 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:07.223 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:07.224 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:07.224 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:07.229 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.230 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:07.231 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:07.233 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.235 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:36:07.245 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:07.246 INFO 4486 --- [ocp-schedules-18,86149e3b5bc6000e,af4df33e3185bf5c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:36:07.254 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:36:07.260 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:36:07.264 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:36:07.266 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:36:07.272 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:36:07.276 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:36:07.282 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:07.284 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:36:07.289 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:07.294 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:07.296 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:36:07.312 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:07.318 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:36:07.321 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:36:07.330 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:36:07.332 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:36:07.346 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:36:07.347 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:07.350 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:07.357 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:36:07.365 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:36:07.374 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:07.375 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:36:07.380 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:36:07.386 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:36:07.390 INFO 4486 --- [host-check-host-status3,28a6d0a2b8fb90e9,4fb952e34ffee4a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:36:07.391 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:07.397 INFO 4486 --- [host-check-host-status4,28a6d0a2b8fb90e9,8de37b7187cf11eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:36:07.410 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:36:07.421 INFO 4486 --- [host-check-host-status2,28a6d0a2b8fb90e9,a452c63ee47555aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:36:07.444 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:36:07.456 INFO 4486 --- [host-check-host-status1,28a6d0a2b8fb90e9,5cd902ab60e3a87a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:36:07.994 INFO 4486 --- [prometheus-minute-schedule-1,480bfcda4b3be395,6fe971b24b85758e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:08.000 INFO 4486 --- [prometheus-minute-schedule-1,480bfcda4b3be395,6fe971b24b85758e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:08.065 INFO 4486 --- [host-check-clockdiff3,2b95f45cbea5c001,dc252b443b1d011b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056968 3 3 , error= 2024-10-16 13:36:08.067 INFO 4486 --- [host-check-clockdiff4,2b95f45cbea5c001,b7f8ab40561c7c4e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056968 3 3 , error= 2024-10-16 13:36:08.068 INFO 4486 --- [host-check-clockdiff2,2b95f45cbea5c001,408253939197bc1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056968 3 3 , error= 2024-10-16 13:36:09.080 INFO 4486 --- [ocp-async-8,301dbb18d9e45c3b,b9e2344315d010ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:09.422 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7b52c4f4ee451b9b,bd966b2a605d4039] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7b52c4f4ee451b9b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:09.422 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7b52c4f4ee451b9b,bd966b2a605d4039] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7b52c4f4ee451b9b, duration=2 ms] 2024-10-16 13:36:09.786 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:10.005 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:36:10.005 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:36:10.027 INFO 4486 --- [Check obproxy1,92fdeed84e3bedaa,ab63f5e55dfdbc73] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:36:10.027 INFO 4486 --- [Check obproxy1,92fdeed84e3bedaa,ab63f5e55dfdbc73] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:10.028 INFO 4486 --- [Check obproxy2,a2885045694fb0f4,9f97b103ed577cd6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:36:10.028 INFO 4486 --- [Check obproxy2,a2885045694fb0f4,9f97b103ed577cd6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:10.030 INFO 4486 --- [Collect obproxy2,2102d656b9266c43,247f533b5d9c945f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:36:10.030 INFO 4486 --- [Collect obproxy2,2102d656b9266c43,247f533b5d9c945f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:10.031 INFO 4486 --- [Collect obproxy1,3b66a06b0e2db310,84b44ff85afdab5a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:36:10.031 INFO 4486 --- [Collect obproxy1,3b66a06b0e2db310,84b44ff85afdab5a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:10.050 INFO 4486 --- [Check obproxy2,a2885045694fb0f4,9f97b103ed577cd6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:36:10.050 INFO 4486 --- [Check obproxy2,a2885045694fb0f4,9f97b103ed577cd6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:36:10.051 INFO 4486 --- [Collect obproxy2,2102d656b9266c43,247f533b5d9c945f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:36:10.052 INFO 4486 --- [Check obproxy1,92fdeed84e3bedaa,ab63f5e55dfdbc73] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:36:10.052 INFO 4486 --- [Check obproxy1,92fdeed84e3bedaa,ab63f5e55dfdbc73] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:36:10.054 INFO 4486 --- [Collect obproxy1,3b66a06b0e2db310,84b44ff85afdab5a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:36:10.056 INFO 4486 --- [Collect obproxy2,2102d656b9266c43,247f533b5d9c945f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:36:10.059 INFO 4486 --- [Collect obproxy1,3b66a06b0e2db310,84b44ff85afdab5a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:36:10.060 INFO 4486 --- [Collect obproxy2,2102d656b9266c43,247f533b5d9c945f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:36:10.063 INFO 4486 --- [Collect obproxy1,3b66a06b0e2db310,84b44ff85afdab5a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:36:10.390 INFO 4486 --- [ocp-schedules-17,360e663d4d015dc3,a3d2bede88883911] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:36:10.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f11ed6cec586fe74,402f1c5aa7780fe8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f11ed6cec586fe74, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:10.962 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f11ed6cec586fe74,402f1c5aa7780fe8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f11ed6cec586fe74, duration=2 ms] 2024-10-16 13:36:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:36:10.994 INFO 4486 --- [prometheus-minute-schedule-1,774d9efbba821247,a1f06f8f3725982b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:11.002 INFO 4486 --- [prometheus-minute-schedule-1,774d9efbba821247,a1f06f8f3725982b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:36:11.080 INFO 4486 --- [ocp-async-6,010bfe009e685444,6e3babd12fe4f087] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:11.307 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b74098649851a79e,45a80c202b38df25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b74098649851a79e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:11.307 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b74098649851a79e,45a80c202b38df25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b74098649851a79e, duration=1 ms] 2024-10-16 13:36:12.390 INFO 4486 --- [ocp-schedules-7,fa5c70c8385e22ab,b0a111ade1807090] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:36:12.417 WARN 4486 --- [scanner-13256303-1,fa5c70c8385e22ab,4e88d8af88a3ae6f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:36:12.513 INFO 4486 --- [alarm-detect-10,fa5c70c8385e22ab,cc074beb018e920c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056945 ~ 1729056962 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056960, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056960, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056960=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:36:12.787 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:12.991 INFO 4486 --- [ocp-async-2,c80b79f280f72454,ab5bc13c09bae6c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:36:13.081 INFO 4486 --- [ocp-async-7,302b7d49cfd3980b,aae3736d3cefbd49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:13.286 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056900, nodeCount=1000, affectRows=1000 2024-10-16 13:36:13.294 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056900, nodeCount=998, validCount=998 2024-10-16 13:36:13.305 ERROR 4486 --- [metric-archive-second-27,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990814) Duplicate entry '586774769061924864-1729056900' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:36:13.303671] [YB420AC531B8-00060B7DB4195AD3-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990814) Duplicate entry '586774769061924864-1729056900' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:36:13.303671] [YB420AC531B8-00060B7DB4195AD3-0-0] 2024-10-16 13:36:13.306 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056900, nodeCount=1000, affectRows=1000 2024-10-16 13:36:13.309 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056900, nodeCount=1000, affectRows=1000 2024-10-16 13:36:13.310 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056900, nodeCount=1000, validCount=1000 2024-10-16 13:36:13.313 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056900, nodeCount=1000, validCount=1000 2024-10-16 13:36:13.386 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5f28098d9a509167,38b7a70e9c2e22b8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=5f28098d9a509167, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:36:13.387 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5f28098d9a509167,38b7a70e9c2e22b8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=5f28098d9a509167, duration=3 ms] 2024-10-16 13:36:13.557 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056900, nodeCount=450, affectRows=450 2024-10-16 13:36:13.558 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056900, nodeCount=449, validCount=449 2024-10-16 13:36:13.994 INFO 4486 --- [prometheus-minute-schedule-1,efac3720f403a163,3bce731ca8f3f493] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:14.001 INFO 4486 --- [prometheus-minute-schedule-1,efac3720f403a163,3bce731ca8f3f493] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:14.438 INFO 4486 --- [metric-archive-async-27,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056900, nodeCount=234, affectRows=234 2024-10-16 13:36:14.454 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056900, nodeCount=243, affectRows=243 2024-10-16 13:36:14.764 INFO 4486 --- [metric-archive-async-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056900, nodeCount=234, validCount=234 2024-10-16 13:36:14.778 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056900, nodeCount=243, validCount=243 2024-10-16 13:36:15.081 INFO 4486 --- [ocp-async-9,254c065ec52c6641,5ef336246d833e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:15.788 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:16.994 INFO 4486 --- [prometheus-minute-schedule-1,b1fc52bc39664da7,90010a8517e654a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:17.000 INFO 4486 --- [prometheus-minute-schedule-1,b1fc52bc39664da7,90010a8517e654a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:17.069 INFO 4486 --- [ocp-schedules-20,1c1e5f226a0ac0d9,762eb6ff721b3e08] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:36:17.081 INFO 4486 --- [ocp-async-3,bafc7b6ddda3c861,b74d49179ba2c59f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:17.105 INFO 4486 --- [ocp-schedules-29,09034a50d462929c,ed38121a597b01a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:36:17.198 INFO 4486 --- [ocp-schedules-21,796dd645470eec7e,c595a69a49fbd7d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:36:17.210 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,07bf495bec7dab3f,2d2720a70a9b7450] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=07bf495bec7dab3f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:17.210 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,07bf495bec7dab3f,2d2720a70a9b7450] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=07bf495bec7dab3f, duration=1 ms] 2024-10-16 13:36:18.789 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:18.897 INFO 4486 --- [ocp-schedules-24,cf440467159adc39,52e013f3720de3c3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:36:19.082 INFO 4486 --- [ocp-async-2,09b3345ce126c28d,65e881c12c23bd22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:19.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,38c4a0046c3e692f,d5c0c21af54e6092] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=38c4a0046c3e692f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:19.428 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,38c4a0046c3e692f,d5c0c21af54e6092] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=38c4a0046c3e692f, duration=2 ms] 2024-10-16 13:36:19.994 INFO 4486 --- [prometheus-minute-schedule-1,3e3bba2dc986a4e0,0a885eecddda9250] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:19.999 INFO 4486 --- [prometheus-minute-schedule-1,3e3bba2dc986a4e0,0a885eecddda9250] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:20.396 INFO 4486 --- [ocp-schedules-1,84e308018a96a4e1,f5619533adaa6d3b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:36:20.811 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:36:20.968 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,68e61a5ce64bc504,c64812693b8511be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=68e61a5ce64bc504, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:20.969 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,68e61a5ce64bc504,c64812693b8511be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=68e61a5ce64bc504, duration=1 ms] 2024-10-16 13:36:20.996 INFO 4486 --- [prometheus-cache-monitor1,84562460f2afee6f,573f3bb80c7ec970] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8300} 2024-10-16 13:36:20.996 INFO 4486 --- [prometheus-cache-monitor1,84562460f2afee6f,573f3bb80c7ec970] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:36:21.082 INFO 4486 --- [ocp-async-5,7c2eb3592fe0f473,e0b06e6bd558a89b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:21.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,3c57a20bf87a332d,8e5bce7a266be192] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3c57a20bf87a332d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:21.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,3c57a20bf87a332d,8e5bce7a266be192] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3c57a20bf87a332d, duration=1 ms] 2024-10-16 13:36:21.790 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:22.607 INFO 4486 --- [ocp-schedules-8,73c1ef27ce25ad9a,0d888c6186432fde] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:36:22.644 WARN 4486 --- [scanner-13256656-1,73c1ef27ce25ad9a,6b345e25e1fbe523] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:36:22.837 INFO 4486 --- [ocp-schedules-11,7ee748e1face62fe,3c31978ba78bdd3f] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:36:22.991 INFO 4486 --- [ocp-async-8,69c4cedb0cd12610,868eb037d427bf92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:36:22.994 INFO 4486 --- [prometheus-minute-schedule-1,4d7b12f4951bd7d7,9613f1ce9f327bf7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:23.001 INFO 4486 --- [prometheus-minute-schedule-1,4d7b12f4951bd7d7,9613f1ce9f327bf7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:23.082 INFO 4486 --- [ocp-async-6,4ce4e025856ac498,a3277f30ec4357d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:24.275 INFO 4486 --- [ocp-schedules-6,1b56c1c4174954fa,9224e62629f28bba] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:36:24.417 INFO 4486 --- [ocp-schedules-10,017860f1fb9e9cba,55ba5f61b5387b3e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:36:24.419 INFO 4486 --- [ocp-schedules-10,017860f1fb9e9cba,55ba5f61b5387b3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:24.456 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:36:24.457 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.461 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:36:24.469 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.469 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:36:24.473 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:36:24.474 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.477 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.478 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:36:24.478 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:36:24.485 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.486 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:36:24.493 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.494 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:36:24.494 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.495 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:36:24.503 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:36:24.503 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.504 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.505 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:24.507 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.508 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:36:24.508 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:36:24.514 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.514 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:24.518 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.519 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.519 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:24.519 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:36:24.522 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.522 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:36:24.528 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.529 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:24.530 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.530 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.530 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:36:24.531 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:36:24.541 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.542 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:36:24.544 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.545 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:36:24.545 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.546 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:36:24.553 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.554 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.555 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.555 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.555 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.555 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:36:24.556 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:36:24.556 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.556 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.557 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.558 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.559 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.559 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.560 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.560 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.561 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.562 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.563 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.563 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.564 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.564 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.565 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.565 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:36:24.566 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.566 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.567 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.567 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.568 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.568 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:24.568 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.569 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.570 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.570 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:36:24.576 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.577 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:24.577 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.578 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:36:24.579 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.581 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.583 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.585 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.586 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.587 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:24.587 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.590 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.592 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.594 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.595 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.596 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:24.597 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.597 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.597 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:36:24.598 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.600 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.602 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.604 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.605 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.607 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.608 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:36:24.610 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:36:24.612 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:36:24.612 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:36:24.612 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:36:24.617 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.619 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.621 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.623 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.625 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.626 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.629 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.630 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.632 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.634 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.636 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.637 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:24.637 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.638 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.639 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:36:24.640 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.641 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:36:24.644 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.645 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:36:24.648 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.649 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.649 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:36:24.649 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.649 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:36:24.649 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.649 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:36:24.649 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:36:24.651 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:36:24.653 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:36:24.653 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:36:24.654 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:36:24.678 INFO 4486 --- [ocp-schedules-27,990071e3b717b8c6,cf382ce14c943933] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:36:24.679 INFO 4486 --- [ocp-schedules-27,990071e3b717b8c6,cf382ce14c943933] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:36:24.707 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:24.708 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.709 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:36:24.712 INFO 4486 --- [Collect all tenants major compactions1,990071e3b717b8c6,214d94605a01d13c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:36:24.712 INFO 4486 --- [Collect all tenants major compactions2,990071e3b717b8c6,bd3d4627875aea00] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:36:24.712 INFO 4486 --- [Collect all tenants major compactions3,990071e3b717b8c6,f5021c92ce399fcd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:36:24.713 INFO 4486 --- [Collect all tenants major compactions4,990071e3b717b8c6,efc3a3f2cc015dfa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:36:24.714 INFO 4486 --- [Collect all tenants major compactions5,990071e3b717b8c6,b7b7743a24110230] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:36:24.719 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.719 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:36:24.719 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.719 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:36:24.719 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:36:24.719 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:36:24.723 INFO 4486 --- [Collect all tenants major compactions1,990071e3b717b8c6,214d94605a01d13c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.723 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:36:24.723 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:36:24.723 INFO 4486 --- [Collect all tenants major compactions2,990071e3b717b8c6,bd3d4627875aea00] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.724 INFO 4486 --- [Collect all tenants major compactions2,990071e3b717b8c6,bd3d4627875aea00] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:36:24.724 INFO 4486 --- [Collect all tenants major compactions1,990071e3b717b8c6,214d94605a01d13c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:36:24.728 INFO 4486 --- [Collect all tenants major compactions2,990071e3b717b8c6,bd3d4627875aea00] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:36:24.728 INFO 4486 --- [Collect all tenants major compactions1,990071e3b717b8c6,214d94605a01d13c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:36:24.729 INFO 4486 --- [Collect all tenants major compactions2,990071e3b717b8c6,bd3d4627875aea00] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:36:24.729 INFO 4486 --- [Collect all tenants major compactions1,990071e3b717b8c6,214d94605a01d13c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:36:24.729 INFO 4486 --- [Collect all tenants major compactions3,990071e3b717b8c6,f5021c92ce399fcd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.731 INFO 4486 --- [Collect all tenants major compactions3,990071e3b717b8c6,f5021c92ce399fcd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:36:24.734 INFO 4486 --- [Collect all tenants major compactions3,990071e3b717b8c6,f5021c92ce399fcd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:36:24.734 INFO 4486 --- [Collect all tenants major compactions3,990071e3b717b8c6,f5021c92ce399fcd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:36:24.743 INFO 4486 --- [Collect all tenants major compactions5,990071e3b717b8c6,b7b7743a24110230] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.743 INFO 4486 --- [Collect all tenants major compactions4,990071e3b717b8c6,efc3a3f2cc015dfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:24.744 INFO 4486 --- [Collect all tenants major compactions5,990071e3b717b8c6,b7b7743a24110230] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:36:24.745 INFO 4486 --- [Collect all tenants major compactions4,990071e3b717b8c6,efc3a3f2cc015dfa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:36:24.748 INFO 4486 --- [Collect all tenants major compactions4,990071e3b717b8c6,efc3a3f2cc015dfa] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:36:24.748 INFO 4486 --- [Collect all tenants major compactions4,990071e3b717b8c6,efc3a3f2cc015dfa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:36:24.749 INFO 4486 --- [Collect all tenants major compactions5,990071e3b717b8c6,b7b7743a24110230] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:36:24.749 INFO 4486 --- [Collect all tenants major compactions5,990071e3b717b8c6,b7b7743a24110230] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:36:24.750 INFO 4486 --- [ocp-schedules-27,990071e3b717b8c6,cf382ce14c943933] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:36:24.790 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:25.082 INFO 4486 --- [ocp-async-2,dc2567ad687289a8,10d795a90ffca7b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:25.994 INFO 4486 --- [prometheus-minute-schedule-1,1e7e56bc002d92a3,25edf815aba3b7c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:26.001 INFO 4486 --- [prometheus-minute-schedule-1,1e7e56bc002d92a3,25edf815aba3b7c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:26.753 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-10-16 13:36:26.756 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=observer, size=5 2024-10-16 13:36:26.758 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=obproxy, size=3 2024-10-16 13:36:26.759 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=observer, size=5 2024-10-16 13:36:26.761 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=obproxy, size=3 2024-10-16 13:36:26.762 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=observer, size=5 2024-10-16 13:36:26.763 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=obproxy, size=3 2024-10-16 13:36:26.764 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=observer, size=5 2024-10-16 13:36:26.766 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=obproxy, size=3 2024-10-16 13:36:26.767 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=observer, size=5 2024-10-16 13:36:26.768 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=obproxy, size=3 2024-10-16 13:36:26.769 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=observer, size=5 2024-10-16 13:36:26.770 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=obproxy, size=3 2024-10-16 13:36:26.771 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=observer, size=5 2024-10-16 13:36:26.772 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=obproxy, size=3 2024-10-16 13:36:26.772 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-10-16 13:36:26.772 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=40024 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703036117302710272 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703036117311098880 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703036117273350145 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703036117273350144 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703036117269155843 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703036117269155842 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703036117269155841 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703036117269155840 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703036117281738753 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703036117281738752 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703036117277544450 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703036117277544449 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703036117277544448 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703036117290127360 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703036117298515968 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703036117239795713 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703036117239795712 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703036117235601410 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703036117235601409 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703036117235601408 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703036117248184323 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703036117248184322 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703036117248184321 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703036117248184320 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703036117243990019 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703036117243990018 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703036117243990017 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703036117243990016 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703036117256572930 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703036117256572929 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703036117256572928 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703036117252378625 2024-10-16 13:36:26.847 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703036117252378624 2024-10-16 13:36:26.908 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703036117264961537 2024-10-16 13:36:26.908 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703036117264961536 2024-10-16 13:36:26.908 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703036117260767234 2024-10-16 13:36:26.908 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703036117260767233 2024-10-16 13:36:26.908 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703036117260767232 2024-10-16 13:36:26.970 INFO 4486 --- [ocp-schedules-26,f3df4ecc5f59801b,f6c905cd1292f2d9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:36:26.970+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:36:26.972 INFO 4486 --- [ocp-schedules-26,f3df4ecc5f59801b,f6c905cd1292f2d9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:36:26.974 INFO 4486 --- [ocp-schedules-26,f3df4ecc5f59801b,f6c905cd1292f2d9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:36:26.974 INFO 4486 --- [ocp-schedules-26,f3df4ecc5f59801b,f6c905cd1292f2d9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703036117206241281 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703036117206241280 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703036117202046977 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703036117202046976 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703036117214629889 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703036117214629888 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703036117223018497 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703036117223018496 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703036117231407106 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703036117231407105 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703036117231407104 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703036117227212801 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703036117227212800 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703036117181075456 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703036117176881152 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703036117189464064 2024-10-16 13:36:27.015 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703036117193658368 2024-10-16 13:36:27.071 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703036117164298240 2024-10-16 13:36:27.083 INFO 4486 --- [ocp-async-5,795b380ccf4b6c10,ee9df4e34d4b876d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:27.217 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d63f32dd3d46f581,8c577b92012744a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d63f32dd3d46f581, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:27.217 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d63f32dd3d46f581,8c577b92012744a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d63f32dd3d46f581, duration=2 ms] 2024-10-16 13:36:27.791 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:28.994 INFO 4486 --- [prometheus-minute-schedule-1,38e51884eef3e024,7931b4b41d42c2e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:29.001 INFO 4486 --- [prometheus-minute-schedule-1,38e51884eef3e024,7931b4b41d42c2e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:29.083 INFO 4486 --- [ocp-async-9,6e50017af19550a0,ebfa355d615851af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:29.144 INFO 4486 --- [ocp-schedules-28,7effc9b92dd0953d,1938e422a03363ef] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=39968 2024-10-16 13:36:29.436 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1faf8223c9e2a165,fab32ecbeb9d8f7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1faf8223c9e2a165, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:29.436 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1faf8223c9e2a165,fab32ecbeb9d8f7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1faf8223c9e2a165, duration=3 ms] 2024-10-16 13:36:30.103 INFO 4486 --- [prometheus-second-schedule-1,43806f52690c9e1a,787832c6701354b8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:36:30.402 INFO 4486 --- [ocp-schedules-17,e47b501fe05d8eaa,25e8890ad6b2618d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:36:30.792 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:30.977 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,18a3a30c97c0d6e0,be4cdf060639b1ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=18a3a30c97c0d6e0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:30.977 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,18a3a30c97c0d6e0,be4cdf060639b1ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=18a3a30c97c0d6e0, duration=1 ms] 2024-10-16 13:36:31.083 INFO 4486 --- [ocp-async-6,488bd77ffb79f9f1,345f3d248924c848] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:31.316 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,76bb2298070e8a37,b16560d7869e0d7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=76bb2298070e8a37, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:31.316 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,76bb2298070e8a37,b16560d7869e0d7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=76bb2298070e8a37, duration=1 ms] 2024-10-16 13:36:31.994 INFO 4486 --- [prometheus-minute-schedule-1,032e31bd62c21a14,9e8cc40ed9bcf948] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:32.000 INFO 4486 --- [prometheus-minute-schedule-1,032e31bd62c21a14,9e8cc40ed9bcf948] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:32.740 INFO 4486 --- [ocp-schedules-12,22bf637ccbaaf19c,88a1d98906756e3d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:36:32.764 WARN 4486 --- [scanner-13256854-1,22bf637ccbaaf19c,ff87af690771b1df] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:36:32.843 INFO 4486 --- [alarm-detect-3,22bf637ccbaaf19c,c9d0cf7f1ff406a7] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056960 ~ 1729056982 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056975, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056975, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056975=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:36:32.992 INFO 4486 --- [ocp-async-2,51bf7f3b6a436485,a773b0c5d49c61d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:36:33.084 INFO 4486 --- [ocp-async-1,3e96c7e23024cd33,3d8e4ec7f3a95972] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:33.793 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:34.580 INFO 4486 --- [ocp-schedules-16,b461b1e5570bfb69,9d63003d89cbca01] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:36:34.581 INFO 4486 --- [ocp-schedules-16,f6a476b989d1675b,4ef87830ce892d38] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:36:34.582 INFO 4486 --- [ocp-schedules-16,f6a476b989d1675b,4ef87830ce892d38] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:34.608 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.609 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:34.620 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.620 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:34.642 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.642 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:34.643 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:34.643 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.643 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:36:34.673 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:34.674 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.675 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:34.676 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:36:34.676 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.679 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:36:34.680 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:34.689 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:34.689 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:34.699 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:34.708 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:34.709 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.709 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:34.710 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:36:34.716 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:36:34.721 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:34.722 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:34.744 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:34.746 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.747 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:36:34.747 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:36:34.775 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:36:34.775 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:36:34.805 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:34.807 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.808 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:36:34.811 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.815 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:36:34.818 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:34.832 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:36:34.844 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:34.845 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:34.845 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:36:34.847 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:34.847 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:34.855 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:36:34.855 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:36:34.878 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:34.878 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:34.906 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:34.906 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:34.941 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:34.941 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:36:34.974 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:34.974 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:36:34.994 INFO 4486 --- [prometheus-minute-schedule-1,ea9de34c1e8f6554,17b61b5a89135cc5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:35.001 INFO 4486 --- [prometheus-minute-schedule-1,ea9de34c1e8f6554,17b61b5a89135cc5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:35.007 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:36:35.036 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:35.038 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:35.039 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:35.043 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:36:35.049 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:36:35.053 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:36:35.055 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:36:35.080 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:36:35.083 INFO 4486 --- [ocp-async-8,e4eafeddd74ad314,408e3a3a24bce473] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:35.225 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:36:35.252 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:35.254 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:35.255 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:36:35.289 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:36:35.289 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:36:36.794 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:37.063 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:36:37.064 INFO 4486 --- [ocp-schedules-22,042672c8104d5dd8,38ebd6d30d7589a3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:36:37.066 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:36:37.067 INFO 4486 --- [ocp-schedules-11,53899ac8b62fa6ab,0014947e2f0b6664] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:36:37.067 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:36:37.069 INFO 4486 --- [ocp-schedules-11,53899ac8b62fa6ab,0014947e2f0b6664] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:37.070 INFO 4486 --- [ocp-schedules-22,042672c8104d5dd8,38ebd6d30d7589a3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:36:37.081 INFO 4486 --- [ocp-schedules-28,da3c572cd0babd20,a146d05d26b17adf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:36:37.083 INFO 4486 --- [ocp-schedules-28,da3c572cd0babd20,a146d05d26b17adf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:37.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:36:37.083 INFO 4486 --- [ocp-schedules-26,f5c588951c086ef9,f85e830a5fffc2c2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 13:36:37.084 INFO 4486 --- [ocp-async-4,96e36990597643d3,13e66b3c114a5bbf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:37.086 INFO 4486 --- [ocp-schedules-26,f5c588951c086ef9,f85e830a5fffc2c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:37.090 INFO 4486 --- [ocp-schedules-20,7704e470fd8f0c7e,c4751cbeb4ed3200] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:36:37.094 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:36:37.093 INFO 4486 --- [ocp-schedules-6,83d9fd896da01b69,e85bcb3a8012405b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:36:37.095 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:36:37.095 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:36:37.095 INFO 4486 --- [ocp-schedules-17,5f256f8112c19f90,7f9bddca1e50e2b2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:36:37.095 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:36:37.097 INFO 4486 --- [host-check-clockdiff1,83d9fd896da01b69,bc633025a45d72b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:36:37.096 INFO 4486 --- [host-check-clockdiff2,83d9fd896da01b69,e06d4767763bd79b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:36:37.097 INFO 4486 --- [host-check-clockdiff3,83d9fd896da01b69,2515d93e9f59bd8f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:36:37.101 INFO 4486 --- [host-check-agent-status1,5f256f8112c19f90,18d5b2932b5c4438] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:36:37.102 INFO 4486 --- [host-check-clockdiff4,83d9fd896da01b69,1806235286b0d39e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:36:37.103 INFO 4486 --- [host-check-agent-status2,5f256f8112c19f90,f9239c97c1fdd0ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:36:37.106 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:37.107 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:36:37.107 INFO 4486 --- [host-check-agent-status3,5f256f8112c19f90,d360c2ef59f4d528] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:36:37.108 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:36:37.108 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:36:37.109 INFO 4486 --- [host-check-agent-status4,5f256f8112c19f90,1b2ca9e3816ac1e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:36:37.111 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:36:37.111 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:36:37.111 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:36:37.111 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:37.133 INFO 4486 --- [host-check-agent-status1,5f256f8112c19f90,18d5b2932b5c4438] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:37.138 INFO 4486 --- [host-check-agent-status1,5f256f8112c19f90,18d5b2932b5c4438] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:37.138 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:37.139 INFO 4486 --- [host-check-clockdiff1,83d9fd896da01b69,bc633025a45d72b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056997 0 0 , error= 2024-10-16 13:36:37.139 INFO 4486 --- [host-check-clockdiff2,83d9fd896da01b69,e06d4767763bd79b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056997 3 3 , error= 2024-10-16 13:36:37.142 INFO 4486 --- [host-check-clockdiff4,83d9fd896da01b69,1806235286b0d39e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056997 3 3 , error= 2024-10-16 13:36:37.145 INFO 4486 --- [host-check-agent-status2,5f256f8112c19f90,f9239c97c1fdd0ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:36:37.148 INFO 4486 --- [host-check-clockdiff3,83d9fd896da01b69,2515d93e9f59bd8f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729056997 3 3 , error= 2024-10-16 13:36:37.148 INFO 4486 --- [host-check-agent-status2,5f256f8112c19f90,f9239c97c1fdd0ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:36:37.148 INFO 4486 --- [host-check-agent-status3,5f256f8112c19f90,d360c2ef59f4d528] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:36:37.149 INFO 4486 --- [host-check-agent-status4,5f256f8112c19f90,1b2ca9e3816ac1e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:36:37.149 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:37.150 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:36:37.150 INFO 4486 --- [host-check-agent-status4,5f256f8112c19f90,1b2ca9e3816ac1e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:36:37.151 INFO 4486 --- [host-check-agent-status3,5f256f8112c19f90,d360c2ef59f4d528] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:36:37.158 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.159 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:36:37.169 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:37.170 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:37.170 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2e418d624cba23e9,1846aa60ee0fa7be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=2e418d624cba23e9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 13:36:37.171 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2e418d624cba23e9,1846aa60ee0fa7be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=2e418d624cba23e9, duration=2 ms] 2024-10-16 13:36:37.171 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:36:37.171 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.172 INFO 4486 --- [host-check-agent-status1,5f256f8112c19f90,18d5b2932b5c4438] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:36:37.172 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.172 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.172 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.173 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:36:37.173 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:36:37.173 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:36:37.173427881+08:00 2024-10-16 13:36:37.174 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.174 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:36:37.174 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:36:37.174 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:36:37.174+08:00 2024-10-16 13:36:37.175 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.175 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:36:37.175 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.175 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:36:37.176 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:36:37.176 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:36:37.177 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:36:37.174+08:00 and status AVAILABLE 2024-10-16 13:36:37.177 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:36:37.180492241+08:00 2024-10-16 13:36:37.178 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:36:37.180939918+08:00 2024-10-16 13:36:37.178 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:36:37.178 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:36:37.178+08:00 2024-10-16 13:36:37.178 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:36:37.178 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:36:37.178+08:00 2024-10-16 13:36:37.178 INFO 4486 --- [host-check-agent-status1,5f256f8112c19f90,18d5b2932b5c4438] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:36:37.178 INFO 4486 --- [host-check-agent-status1,5f256f8112c19f90,18d5b2932b5c4438] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:36:37.178 INFO 4486 --- [host-check-agent-status1,5f256f8112c19f90,18d5b2932b5c4438] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:36:37.180 INFO 4486 --- [host-check-agent-status4,5f256f8112c19f90,1b2ca9e3816ac1e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:36:37.181 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:36:37.183642924+08:00 2024-10-16 13:36:37.181 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:36:37.181 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:36:37.178+08:00 and status AVAILABLE 2024-10-16 13:36:37.181 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:36:37.178+08:00 and status AVAILABLE 2024-10-16 13:36:37.181 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:36:37.181+08:00 2024-10-16 13:36:37.181 INFO 4486 --- [host-check-agent-status2,5f256f8112c19f90,f9239c97c1fdd0ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:36:37.181 INFO 4486 --- [host-check-agent-status3,5f256f8112c19f90,d360c2ef59f4d528] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:36:37.184 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:36:37.181+08:00 and status AVAILABLE 2024-10-16 13:36:37.184 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:36:37.184 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.184 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:36:37.185 INFO 4486 --- [host-check-agent-status4,5f256f8112c19f90,1b2ca9e3816ac1e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:36:37.185 INFO 4486 --- [host-check-agent-status4,5f256f8112c19f90,1b2ca9e3816ac1e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:36:37.185 INFO 4486 --- [host-check-agent-status4,5f256f8112c19f90,1b2ca9e3816ac1e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:36:37.185 INFO 4486 --- [host-check-agent-status3,5f256f8112c19f90,d360c2ef59f4d528] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:36:37.185 INFO 4486 --- [host-check-agent-status3,5f256f8112c19f90,d360c2ef59f4d528] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:36:37.185 INFO 4486 --- [host-check-agent-status3,5f256f8112c19f90,d360c2ef59f4d528] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:36:37.186 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.186 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:36:37.187 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:37.187 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 13:36:37.188 INFO 4486 --- [host-check-agent-status2,5f256f8112c19f90,f9239c97c1fdd0ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:36:37.188 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.188 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.188 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.188 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.188 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.188 INFO 4486 --- [host-check-agent-status2,5f256f8112c19f90,f9239c97c1fdd0ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:36:37.188 INFO 4486 --- [host-check-agent-status2,5f256f8112c19f90,f9239c97c1fdd0ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:36:37.189 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.189 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.190 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:36:37.192 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:37.192 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:36:37.193 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:36:37.194 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:37.194 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:36:37.194 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:36:37.195 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:36:37.195 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:36:37.198 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.199 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:37.202 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.205 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:36:37.206 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 13:36:37.208 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.209 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.209 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:36:37.211 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.213 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 13:36:37.214 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:37.214 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:37.216 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:36:37.216 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.216 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:37.218 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:36:37.220 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 13:36:37.222 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5f3fcebc9c77b65d,54dcb21271efea5b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5f3fcebc9c77b65d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:37.222 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5f3fcebc9c77b65d,54dcb21271efea5b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5f3fcebc9c77b65d, duration=1 ms] 2024-10-16 13:36:37.228 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.229 INFO 4486 --- [ocp-schedules-7,35aa40e665056435,7f10d3214879960e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:36:37.247 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:36:37.248 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:36:37.256 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:36:37.257 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:36:37.259 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:36:37.269 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:36:37.272 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:36:37.278 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:37.280 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:37.283 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:36:37.288 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:37.298 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:36:37.306 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:36:37.315 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:36:37.316 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:36:37.325 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:36:37.326 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:36:37.332 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:37.337 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:36:37.343 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:37.350 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:36:37.353 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:37.359 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:36:37.367 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:36:37.374 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:36:37.375 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:36:37.377 INFO 4486 --- [host-check-host-status3,7704e470fd8f0c7e,d99dad4fe11be961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:36:37.384 INFO 4486 --- [host-check-host-status4,7704e470fd8f0c7e,4e2606ec7afc4171] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:36:37.390 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:36:37.399 INFO 4486 --- [host-check-host-status2,7704e470fd8f0c7e,6d51e33ec1d0ac15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:36:37.405 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:37.408 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.409 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:36:37.410 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.411 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.411 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:36:37.422 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.422 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.422 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.422 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.422 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.422 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.432 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:36:37.445 INFO 4486 --- [host-check-host-status1,7704e470fd8f0c7e,7d75358a2fc641e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:36:37.451 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:36:37.483 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:36:37.529 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:37.544 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:37.545 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.546 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:36:37.547 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:36:37.548 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.549 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:36:37.550 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.550 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.551 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:36:37.561 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.561 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.561 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.561 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.561 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.562 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.569 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:36:37.761 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:36:37.803 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:37.805 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.808 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:36:37.809 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.809 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.810 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:36:37.820 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.820 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.820 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.820 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.820 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:37.821 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:37.838 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:36:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:36:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:36:37.994 INFO 4486 --- [prometheus-minute-schedule-1,997f897369b35ae1,f8cd6018103bb7dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:37.995 INFO 4486 --- [ocp-schedules-3,cbd07ce018adb355,3b56c060430fe46d] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 13:36:37.995 INFO 4486 --- [ocp-schedules-3,cbd07ce018adb355,3b56c060430fe46d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 13:36:38.001 INFO 4486 --- [prometheus-minute-schedule-1,997f897369b35ae1,f8cd6018103bb7dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:38.031 INFO 4486 --- [pool-10-thread-16,8d9e76f07f68e702,21e82e98d3c67ec0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 13:36:38.033 INFO 4486 --- [pool-10-thread-13,8d9e76f07f68e702,c558982ac8d9723a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 13:36:38.033 INFO 4486 --- [pool-10-thread-13,8d9e76f07f68e702,c558982ac8d9723a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 13:36:38.033 INFO 4486 --- [pool-10-thread-15,8d9e76f07f68e702,30d9b2db1d7cc823] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 13:36:38.051 INFO 4486 --- [pool-10-thread-14,8d9e76f07f68e702,417fcc0837ed7175] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 13:36:38.054 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 13:36:38.054 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 13:36:38.055 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 13:36:38.055 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 13:36:38.059 INFO 4486 --- [ForkJoinPool-6-worker-6,8d9e76f07f68e702,872a8d067b271c3f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 13:36:38.059 INFO 4486 --- [ForkJoinPool-6-worker-6,8d9e76f07f68e702,872a8d067b271c3f] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 13:36:38.060 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:36:38.060 INFO 4486 --- [pool-overview-container-5,61b279af93643396,23c7b75392f936ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.061 INFO 4486 --- [pool-10-thread-14,8d9e76f07f68e702,417fcc0837ed7175] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 13:36:38.061 INFO 4486 --- [pool-10-thread-14,8d9e76f07f68e702,417fcc0837ed7175] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 13:36:38.063 INFO 4486 --- [pool-overview-container-5,61b279af93643396,23c7b75392f936ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:36:38.081 INFO 4486 --- [ForkJoinPool-6-worker-9,8d9e76f07f68e702,541e6665f5729ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.081 INFO 4486 --- [ForkJoinPool-6-worker-6,8d9e76f07f68e702,872a8d067b271c3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.082 INFO 4486 --- [ForkJoinPool-6-worker-6,8d9e76f07f68e702,872a8d067b271c3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:36:38.083 INFO 4486 --- [ForkJoinPool-6-worker-9,8d9e76f07f68e702,541e6665f5729ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:36:38.083 INFO 4486 --- [ForkJoinPool-6-worker-6,8d9e76f07f68e702,872a8d067b271c3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.084 INFO 4486 --- [ForkJoinPool-6-worker-6,8d9e76f07f68e702,872a8d067b271c3f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:36:38.087 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.088 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:36:38.090 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.093 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:36:38.094 INFO 4486 --- [ForkJoinPool-6-worker-9,8d9e76f07f68e702,541e6665f5729ca2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 13:36:38.094 INFO 4486 --- [ForkJoinPool-6-worker-9,8d9e76f07f68e702,541e6665f5729ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.095 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.095 INFO 4486 --- [ForkJoinPool-6-worker-9,8d9e76f07f68e702,541e6665f5729ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:36:38.095 INFO 4486 --- [pool-overview-container-5,61b279af93643396,23c7b75392f936ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.096 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:36:38.097 INFO 4486 --- [ForkJoinPool-6-worker-9,8d9e76f07f68e702,541e6665f5729ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.098 INFO 4486 --- [ForkJoinPool-6-worker-9,8d9e76f07f68e702,541e6665f5729ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:36:38.098 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.099 INFO 4486 --- [pool-overview-container-5,61b279af93643396,23c7b75392f936ec] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:36:38.099 INFO 4486 --- [ForkJoinPool-6-worker-2,8d9e76f07f68e702,75e2ebe5db696c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.106 WARN 4486 --- [ForkJoinPool-6-worker-6,8d9e76f07f68e702,872a8d067b271c3f] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:36:38.106 INFO 4486 --- [pool-10-thread-13,8d9e76f07f68e702,c558982ac8d9723a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 13:36:38.099 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:36:38.108 INFO 4486 --- [ForkJoinPool-6-worker-2,8d9e76f07f68e702,75e2ebe5db696c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:36:38.111 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.112 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:36:38.114 INFO 4486 --- [pool-10-thread-13,8d9e76f07f68e702,c558982ac8d9723a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 13:36:38.118 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:36:38.120 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.120 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:38.122 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.122 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.123 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:36:38.123 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:36:38.125 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:36:38.125 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:38.125 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.127 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:36:38.128 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.129 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:36:38.134 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:38.134 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:38.134 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:38.134 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:38.134 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:36:38.135 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.136 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.136 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:36:38.144 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.145 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:36:38.146 INFO 4486 --- [ForkJoinPool-6-worker-2,8d9e76f07f68e702,75e2ebe5db696c64] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:36:38.147 INFO 4486 --- [ForkJoinPool-6-worker-2,8d9e76f07f68e702,75e2ebe5db696c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.149 INFO 4486 --- [ForkJoinPool-6-worker-2,8d9e76f07f68e702,75e2ebe5db696c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:36:38.154 INFO 4486 --- [ForkJoinPool-6-worker-2,8d9e76f07f68e702,75e2ebe5db696c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.155 INFO 4486 --- [ForkJoinPool-6-worker-2,8d9e76f07f68e702,75e2ebe5db696c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:36:38.159 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.160 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.161 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:38.161 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:36:38.165 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:36:38.167 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.168 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:36:38.169 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.175 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:36:38.176 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.176 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.178 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.180 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:36:38.181 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:36:38.181 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:36:38.182 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:36:38.183 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.185 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:36:38.187 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.187 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.188 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:36:38.189 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:36:38.193 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 13:36:38.201 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:36:38.208 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:36:38.211 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.212 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:36:38.217 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.217 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.217 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.217 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.218 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:36:38.219 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:36:38, 2024-10-16 13:36:38] 2024-10-16 13:36:38.220 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:36:38.222 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 13:36:38.223 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 13:36:38.226 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.232 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:36:38.235 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:38.236 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.237 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:36:38.239 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.239 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:36:38.240 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.241 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:36:38.242 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:36:38, 2024-10-16 13:36:38] 2024-10-16 13:36:38.242 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:36:38.243 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.244 INFO 4486 --- [ForkJoinPool-6-worker-0,8d9e76f07f68e702,ada1a442733a66bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:36:38.246 INFO 4486 --- [pool-10-thread-16,8d9e76f07f68e702,21e82e98d3c67ec0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 13:36:38.250 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 13:36:38.252 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.254 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:36:38, 2024-10-16 13:36:38] 2024-10-16 13:36:38.263 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:36:38.265 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.267 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:36:38, 2024-10-16 13:36:38] 2024-10-16 13:36:38.269 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:38.270 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.271 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:38.272 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.276 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 13:36:38.278 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 13:36:38.278 INFO 4486 --- [ForkJoinPool-6-worker-11,8d9e76f07f68e702,e0fa88f58be12a1e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 13:36:38.278 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:36:38.285 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 13:36:38.287 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.289 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:36:38, 2024-10-16 13:36:38] 2024-10-16 13:36:38.317 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:36:38.319 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.320 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:36:38.321 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:36:38.322 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:36:38, 2024-10-16 13:36:38] 2024-10-16 13:36:38.326 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 13:36:38.326 INFO 4486 --- [ForkJoinPool-6-worker-4,8d9e76f07f68e702,882f90d8eb1eec74] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 13:36:38.329 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:36:38.329 INFO 4486 --- [pool-10-thread-15,8d9e76f07f68e702,30d9b2db1d7cc823] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 13:36:38.424 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:36:38.204351, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:36:38.425 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:36:38.204351, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:36:38.461 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:36:38.204351} } 2024-10-16 13:36:38.519 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:36:38.526 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:36:38.529 INFO 4486 --- [ocp-schedules-5,74d0b2fe7229144f,1cb546a247ba2791] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:36:38.902 INFO 4486 --- [ocp-schedules-19,cf0a5c94781de5c7,cfe87b54760eef36] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:36:39.084 INFO 4486 --- [ocp-async-2,a2cb5508f5551bec,dd3ad3671158ee77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:39.442 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,164ca04f3ef7c163,424d41b9df303090] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=164ca04f3ef7c163, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:39.442 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,164ca04f3ef7c163,424d41b9df303090] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=164ca04f3ef7c163, duration=1 ms] 2024-10-16 13:36:39.795 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:40.007 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:36:40.008 INFO 4486 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:36:40.028 INFO 4486 --- [Check obproxy2,faa57a022069b760,08e9c5984214d927] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:36:40.029 INFO 4486 --- [Check obproxy2,faa57a022069b760,08e9c5984214d927] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:40.029 INFO 4486 --- [Check obproxy1,b9e2207e43b7b329,9b2606cf26d5048f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:36:40.029 INFO 4486 --- [Check obproxy1,b9e2207e43b7b329,9b2606cf26d5048f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:40.030 INFO 4486 --- [Collect obproxy2,5a38adbd4160816e,cc2a7564e8748638] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:36:40.031 INFO 4486 --- [Collect obproxy2,5a38adbd4160816e,cc2a7564e8748638] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:40.031 INFO 4486 --- [Collect obproxy1,7a81e9820df37191,db2cbb79f7e6595d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:36:40.031 INFO 4486 --- [Collect obproxy1,7a81e9820df37191,db2cbb79f7e6595d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:36:40.053 INFO 4486 --- [Collect obproxy2,5a38adbd4160816e,cc2a7564e8748638] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:36:40.054 INFO 4486 --- [Check obproxy2,faa57a022069b760,08e9c5984214d927] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:36:40.054 INFO 4486 --- [Check obproxy2,faa57a022069b760,08e9c5984214d927] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:36:40.057 INFO 4486 --- [Check obproxy1,b9e2207e43b7b329,9b2606cf26d5048f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:36:40.057 INFO 4486 --- [Check obproxy1,b9e2207e43b7b329,9b2606cf26d5048f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:36:40.059 INFO 4486 --- [Collect obproxy2,5a38adbd4160816e,cc2a7564e8748638] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:36:40.063 INFO 4486 --- [Collect obproxy1,7a81e9820df37191,db2cbb79f7e6595d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:36:40.063 INFO 4486 --- [Collect obproxy2,5a38adbd4160816e,cc2a7564e8748638] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:36:40.067 INFO 4486 --- [Collect obproxy1,7a81e9820df37191,db2cbb79f7e6595d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:36:40.070 INFO 4486 --- [Collect obproxy1,7a81e9820df37191,db2cbb79f7e6595d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:36:40.407 INFO 4486 --- [ocp-schedules-21,3990f98f7cb023a6,4084952aba79ae3a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:36:40.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b36b52b9b3b4b8d7,fcf6f0d87a0167d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=b36b52b9b3b4b8d7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:36:40.632 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b36b52b9b3b4b8d7,fcf6f0d87a0167d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=b36b52b9b3b4b8d7, duration=2 ms] 2024-10-16 13:36:40.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,1e64f95f50c8faca,f040d36c8c27a12b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1e64f95f50c8faca, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:40.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,1e64f95f50c8faca,f040d36c8c27a12b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1e64f95f50c8faca, duration=1 ms] 2024-10-16 13:36:40.994 INFO 4486 --- [prometheus-minute-schedule-1,ad8b780ce9572400,ab8798d031f13a9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:40.995 INFO 4486 --- [prometheus-hour-schedule-1,3aa73fce851fa642,77d5baeaaedb5766] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:36:41.001 INFO 4486 --- [prometheus-minute-schedule-1,ad8b780ce9572400,ab8798d031f13a9e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:41.084 INFO 4486 --- [ocp-async-5,9e0cff2d9c8cb308,7bc2d91720e17a7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:41.336 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,179c273e1bb8c09a,cb745daeb29d66aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=179c273e1bb8c09a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:41.336 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,179c273e1bb8c09a,cb745daeb29d66aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=179c273e1bb8c09a, duration=1 ms] 2024-10-16 13:36:42.093 INFO 4486 --- [ocp-schedules-15,a632f51bd5808547,c9c85b9f223ae243] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:36:42.796 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:42.926 INFO 4486 --- [ocp-schedules-23,11ef05d89724e9de,95382a53fbfc2ee7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:36:42.993 INFO 4486 --- [ocp-async-9,83c3d472c35bbbb1,d56923b5a724abc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:36:42.994 WARN 4486 --- [scanner-13257235-1,11ef05d89724e9de,6e2cd0e9614981d1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:36:43.083 INFO 4486 --- [alarm-detect-16,11ef05d89724e9de,6399e575f154bf44] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056975 ~ 1729056992 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056990, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729056990, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729056990=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:36:43.084 INFO 4486 --- [ocp-async-3,60f983f95ca2eb97,13d8c463c7799023] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:43.462 INFO 4486 --- [ocp-schedules-1,cab4de7c105ad95d,bc238e81afdd8638] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:36:43.464 INFO 4486 --- [ocp-schedules-1,cab4de7c105ad95d,bc238e81afdd8638] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:36:43.464 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:36:43.465 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:36:43.465 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:36:43.994 INFO 4486 --- [prometheus-minute-schedule-1,f15710b4dab9332c,efcbcb0634be9f4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:44.001 INFO 4486 --- [prometheus-minute-schedule-1,f15710b4dab9332c,efcbcb0634be9f4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:45.084 INFO 4486 --- [ocp-async-6,814dc4ca6f7ac1fb,4525b2e3932a0f80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:45.459 INFO 4486 --- [ocp-schedules-18,89d806691cdf8ef5,5e328a624cee2a88] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:36:45.465 INFO 4486 --- [ocp-schedules-18,89d806691cdf8ef5,5e328a624cee2a88] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:36:45.465 INFO 4486 --- [ocp-schedules-18,89d806691cdf8ef5,5e328a624cee2a88] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:36:45.479 INFO 4486 --- [ocp-schedules-15,e313e92e717548d1,d9e391e5173ddac8] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:36:45.479 INFO 4486 --- [ocp-schedules-15,e313e92e717548d1,d9e391e5173ddac8] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:36:45.797 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:46.994 INFO 4486 --- [prometheus-minute-schedule-1,f7e2014e7de9f273,93f735beb1794a37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:46.999 INFO 4486 --- [prometheus-minute-schedule-1,f7e2014e7de9f273,93f735beb1794a37] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:47.071 INFO 4486 --- [ocp-schedules-11,53899ac8b62fa6ab,0014947e2f0b6664] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:36:47.085 INFO 4486 --- [ocp-async-10,be36befdb2772d54,c58b2fcff1fe7aa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:47.108 INFO 4486 --- [ocp-schedules-25,dc8ab6206c351b41,c91f7f183b0ee515] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:36:47.179 INFO 4486 --- [ocp-schedules-28,da3c572cd0babd20,a146d05d26b17adf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:36:47.223 INFO 4486 --- [ocp-schedules-26,f5c588951c086ef9,f85e830a5fffc2c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 13:36:47.226 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,8b6b504ad1fd1d49,a9840475e2099130] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8b6b504ad1fd1d49, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:47.227 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,8b6b504ad1fd1d49,a9840475e2099130] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8b6b504ad1fd1d49, duration=2 ms] 2024-10-16 13:36:48.798 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:49.085 INFO 4486 --- [ocp-async-4,7df7861d63e724c4,e90caf6749d717fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:49.448 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,b2a858e93d2f61e3,06a7c1d938bc8767] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b2a858e93d2f61e3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:49.448 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,b2a858e93d2f61e3,06a7c1d938bc8767] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b2a858e93d2f61e3, duration=1 ms] 2024-10-16 13:36:49.994 INFO 4486 --- [prometheus-minute-schedule-1,780eebbba5ed7bcb,be09c846c57eaa33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:49.999 INFO 4486 --- [prometheus-minute-schedule-1,780eebbba5ed7bcb,be09c846c57eaa33] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:50.138 INFO 4486 --- [ocp-schedules-3,2155a2e54ff2791f,1eef0c7970001090] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:36:50.138 INFO 4486 --- [ocp-schedules-3,2155a2e54ff2791f,1eef0c7970001090] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:36:50.138 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:36:50.138 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:36:50.138 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:36:50.138 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:36:50.138 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:36:50.144 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:36:50.144 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:36:50.144 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:36:50.144 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:36:50.144 INFO 4486 --- [ocp-schedules-3,2155a2e54ff2791f,1eef0c7970001090] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:36:50.144 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:36:50.145 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:36:50.145 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:36:50.145 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:36:50.145 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:36:50.145 INFO 4486 --- [ocp-schedules-3,2155a2e54ff2791f,1eef0c7970001090] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:36:50.146 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:36:50.170 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:36:50.170 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:36:50.171 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:36:50.179 INFO 4486 --- [ocp-schedules-3,2155a2e54ff2791f,1eef0c7970001090] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:36:50.179 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:36:50.182 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:36:50.187 INFO 4486 --- [ocp-schedules-3,2155a2e54ff2791f,1eef0c7970001090] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:36:50.393 INFO 4486 --- [ocp-schedules-21,988b94b6ff8494b8,bf086ba11a595525] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:36:50.393 INFO 4486 --- [ocp-schedules-21,988b94b6ff8494b8,bf086ba11a595525] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:36:50.394 INFO 4486 --- [ocp-schedules-21,988b94b6ff8494b8,bf086ba11a595525] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:36:50.394 INFO 4486 --- [ocp-schedules-21,988b94b6ff8494b8,bf086ba11a595525] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:36:50.395 INFO 4486 --- [ocp-schedules-21,988b94b6ff8494b8,bf086ba11a595525] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:36:50.395 INFO 4486 --- [ocp-schedules-21,988b94b6ff8494b8,bf086ba11a595525] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703036117311098880 2024-10-16 13:36:50.397 INFO 4486 --- [ocp-schedules-21,988b94b6ff8494b8,bf086ba11a595525] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=56, maxSyncedRecordId=703038633901887488 2024-10-16 13:36:50.415 INFO 4486 --- [ocp-schedules-4,e8e79ac976259aa8,26b98d525ad5c2bd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:36:50.592 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:36:50.593 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:36:50.605 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:36:50.605 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:36:50.605 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:36:50.617 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:36:50.617 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:36:50.617 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:36:50.623 INFO 4486 --- [ocp-schedules-14,3f5006d385c609a9,83c5300da9bf378e] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:36:50.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,9587db4ca079d494,2a3f08fd630ef68d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9587db4ca079d494, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:50.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,9587db4ca079d494,2a3f08fd630ef68d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9587db4ca079d494, duration=1 ms] 2024-10-16 13:36:51.085 INFO 4486 --- [ocp-async-7,0f3952d1285cc644,f4cc241fb5783a74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:51.343 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,e3e73c72b05d7117,bc0dfd84803541e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e3e73c72b05d7117, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:51.343 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,e3e73c72b05d7117,bc0dfd84803541e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e3e73c72b05d7117, duration=1 ms] 2024-10-16 13:36:51.799 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:52.993 INFO 4486 --- [ocp-async-6,067dc1132c77f373,e118f2f8d2c58086] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:36:52.994 INFO 4486 --- [prometheus-minute-schedule-1,4ca98850b5244d55,70a7216163ede353] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:53.001 INFO 4486 --- [prometheus-minute-schedule-1,4ca98850b5244d55,70a7216163ede353] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:53.085 INFO 4486 --- [ocp-async-5,471b6f3f1d3f84c4,13e81b5cb0a6d9b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:53.154 INFO 4486 --- [ocp-schedules-15,5f0ef5bfde09dc6d,149d83ea5fe31a1c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:36:53.178 WARN 4486 --- [scanner-13257448-1,5f0ef5bfde09dc6d,a659b7e2bf23cdca] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:36:53.465 INFO 4486 --- [ocp-schedules-1,cab4de7c105ad95d,bc238e81afdd8638] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:36:54.774 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,554892661da58695,1dd9056e27be3cef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=554892661da58695, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:36:54.774 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,554892661da58695,1dd9056e27be3cef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=554892661da58695, duration=2 ms] 2024-10-16 13:36:54.799 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:55.086 INFO 4486 --- [ocp-async-4,9de81c9370524ee8,76282dcf8f4f432f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:55.994 INFO 4486 --- [prometheus-minute-schedule-1,a0c8d58761b53bc2,12973e4998e12f56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:56.000 INFO 4486 --- [prometheus-minute-schedule-1,a0c8d58761b53bc2,12973e4998e12f56] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:57.086 INFO 4486 --- [ocp-async-7,578fd9b59d4d31d7,1a327a8056ea7470] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:57.234 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d834ae90cff964f6,eac3da43c75fbf34] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d834ae90cff964f6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:57.235 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d834ae90cff964f6,eac3da43c75fbf34] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d834ae90cff964f6, duration=3 ms] 2024-10-16 13:36:57.801 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:36:58.906 INFO 4486 --- [ocp-schedules-23,6457b1520711105a,24879bae5358d833] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:36:58.994 INFO 4486 --- [prometheus-minute-schedule-1,6a0a090999746cdf,26881dfce40de368] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057020, interval=60 2024-10-16 13:36:59.000 INFO 4486 --- [prometheus-minute-schedule-1,6a0a090999746cdf,26881dfce40de368] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:36:59.086 INFO 4486 --- [ocp-async-1,44de476305abb3ec,9f7482d31313e339] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:36:59.455 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ffea8f6f8d7228a6,24b1c3b4c0d7363f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ffea8f6f8d7228a6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:36:59.456 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ffea8f6f8d7228a6,24b1c3b4c0d7363f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ffea8f6f8d7228a6, duration=2 ms] 2024-10-16 13:37:00.421 INFO 4486 --- [ocp-schedules-21,7ad00d395056afce,4165ce80bcd61908] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:37:00.802 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:00.991 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d9e6f00e6d46dd7a,cccfaa5f94820cad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d9e6f00e6d46dd7a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:00.991 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d9e6f00e6d46dd7a,cccfaa5f94820cad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d9e6f00e6d46dd7a, duration=1 ms] 2024-10-16 13:37:01.086 INFO 4486 --- [ocp-async-9,eb005e891830e3ce,9e9ac5341caddcaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:01.188 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,771c824bec60c0b2,9585ba9256b68118] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=771c824bec60c0b2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:37:01.189 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,771c824bec60c0b2,9585ba9256b68118] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=771c824bec60c0b2, duration=2 ms] 2024-10-16 13:37:01.271 INFO 4486 --- [ocp-schedules-12,2ae6cb5e34033500,cf2b691f4c0c0e6b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:37:01.271+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:37:01.275 INFO 4486 --- [ocp-schedules-12,2ae6cb5e34033500,cf2b691f4c0c0e6b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:37:01.285 INFO 4486 --- [ocp-schedules-12,2ae6cb5e34033500,cf2b691f4c0c0e6b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:37:01.296 INFO 4486 --- [ocp-schedules-12,2ae6cb5e34033500,cf2b691f4c0c0e6b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:37:01.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,010fff9ca0d2318c,64fc29285de8e9fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=010fff9ca0d2318c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:01.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,010fff9ca0d2318c,64fc29285de8e9fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=010fff9ca0d2318c, duration=1 ms] 2024-10-16 13:37:01.994 INFO 4486 --- [prometheus-minute-schedule-1,aaa826c4732187fe,b44e1f3ce63c080a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:02.000 INFO 4486 --- [prometheus-minute-schedule-1,aaa826c4732187fe,b44e1f3ce63c080a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:02.993 INFO 4486 --- [ocp-async-4,a8e3d20d6f9e6cef,f040a62efaa9654f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:37:03.087 INFO 4486 --- [ocp-async-2,2a2846ae53fad5d0,36b5922f2ad0abf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:03.336 INFO 4486 --- [ocp-schedules-26,46a0c430e73cf9f9,dc826e68f1debf3d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:37:03.359 WARN 4486 --- [scanner-13257759-1,46a0c430e73cf9f9,18d2ee7232919411] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:37:03.414 INFO 4486 --- [alarm-detect-4,46a0c430e73cf9f9,822b57698d9b1b1e] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729056990 ~ 1729057013 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057005, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057005, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057005=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:37:03.803 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:04.994 INFO 4486 --- [prometheus-minute-schedule-1,9dc1a016a9025732,42fa9207f2eb7656] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:05.000 INFO 4486 --- [prometheus-minute-schedule-1,9dc1a016a9025732,42fa9207f2eb7656] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:05.087 INFO 4486 --- [ocp-async-10,42928576c7d78251,ba8a80b3b2de4e5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:05.290 INFO 4486 --- [ocp-schedules-16,f6a476b989d1675b,4ef87830ce892d38] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:37:05.290 INFO 4486 --- [ocp-schedules-16,7c21456fcca1dcbf,f625ea1fe718dd77] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:37:05.291 INFO 4486 --- [ocp-schedules-16,7c21456fcca1dcbf,f625ea1fe718dd77] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:05.316 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.317 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:05.327 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.328 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:05.351 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:05.352 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.352 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:37:05.355 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.356 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:05.382 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:05.382 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.383 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:05.384 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:37:05.384 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.387 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:37:05.389 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:05.399 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.399 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:05.411 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.419 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:05.420 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.421 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:05.422 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:37:05.426 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:37:05.431 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:05.432 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:05.455 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:37:05.459 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:05.460 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.462 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:37:05.511 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:37:05.511 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:37:05.522 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:05.524 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.525 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:05.529 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.535 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:37:05.538 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:05.548 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:37:05.561 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:05.562 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.563 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:37:05.575 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.575 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:05.576 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:37:05.576 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:37:05.610 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.610 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:05.641 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.641 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:05.673 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.673 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:05.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:05.738 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:05.765 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:05.766 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:05.767 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:05.772 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:37:05.778 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:37:05.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:37:05.785 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:37:05.809 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:37:05.969 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:37:06.000 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:06.002 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:06.003 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:37:06.032 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:37:06.032 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:37:06.804 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:07.066 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:37:07.067 INFO 4486 --- [ocp-schedules-22,6282368eb0569354,267d3b631d4518d1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:37:07.068 INFO 4486 --- [ocp-schedules-22,6282368eb0569354,267d3b631d4518d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:07.081 INFO 4486 --- [ocp-schedules-1,864b5aca05972f62,d9696cae09703fe7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:37:07.083 INFO 4486 --- [ocp-schedules-1,864b5aca05972f62,d9696cae09703fe7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:37:07.087 INFO 4486 --- [ocp-async-8,d76b172b2d9d4997,703d5c599bb6f26e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:07.089 INFO 4486 --- [ocp-schedules-26,1839eb7e19f1632c,52a5a03d9c51922d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:37:07.089 INFO 4486 --- [ocp-schedules-7,a9a81f92651625a6,8f2e39f6d6bb15fd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:37:07.089 INFO 4486 --- [host-check-clockdiff1,1839eb7e19f1632c,c6ad097a12a9336e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:37:07.089 INFO 4486 --- [host-check-clockdiff2,1839eb7e19f1632c,b8fb3058382da60f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:37:07.090 INFO 4486 --- [host-check-clockdiff3,1839eb7e19f1632c,bdbe3c3a68a675bb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:37:07.090 INFO 4486 --- [ocp-schedules-14,9f819d7502a814a5,1ea36ea223965e15] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:37:07.091 INFO 4486 --- [host-check-clockdiff4,1839eb7e19f1632c,005d7ca175ab3d61] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:37:07.091 INFO 4486 --- [host-check-agent-status1,9f819d7502a814a5,27612867c14c3d96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:37:07.091 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:37:07.092 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:37:07.093 INFO 4486 --- [host-check-agent-status2,9f819d7502a814a5,3e2fcd380e3d4c52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:37:07.093 INFO 4486 --- [host-check-agent-status3,9f819d7502a814a5,75a1ec8daeb5bd78] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:37:07.093 INFO 4486 --- [host-check-agent-status4,9f819d7502a814a5,d2eecf7d4aa6f0a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:37:07.093 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:37:07.094 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:37:07.122 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:07.124 INFO 4486 --- [host-check-clockdiff1,1839eb7e19f1632c,c6ad097a12a9336e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057027 0 0 , error= 2024-10-16 13:37:07.126 INFO 4486 --- [host-check-clockdiff3,1839eb7e19f1632c,bdbe3c3a68a675bb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057027 3 3 , error= 2024-10-16 13:37:07.127 INFO 4486 --- [host-check-agent-status4,9f819d7502a814a5,d2eecf7d4aa6f0a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:07.128 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:07.128 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:07.128 INFO 4486 --- [host-check-agent-status3,9f819d7502a814a5,75a1ec8daeb5bd78] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:07.129 INFO 4486 --- [host-check-agent-status4,9f819d7502a814a5,d2eecf7d4aa6f0a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:07.129 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:07.129 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:07.130 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:07.130 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:07.130 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.130 INFO 4486 --- [host-check-agent-status3,9f819d7502a814a5,75a1ec8daeb5bd78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:07.130 INFO 4486 --- [host-check-agent-status2,9f819d7502a814a5,3e2fcd380e3d4c52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:07.130 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:07.131 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:07.132 INFO 4486 --- [host-check-agent-status2,9f819d7502a814a5,3e2fcd380e3d4c52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:07.132 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:37:07.132 INFO 4486 --- [host-check-agent-status1,9f819d7502a814a5,27612867c14c3d96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:07.134 INFO 4486 --- [host-check-agent-status1,9f819d7502a814a5,27612867c14c3d96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:07.153 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.154 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:07.155 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:37:07.156 INFO 4486 --- [host-check-agent-status3,9f819d7502a814a5,75a1ec8daeb5bd78] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:37:07.157 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:37:07.157 INFO 4486 --- [host-check-agent-status4,9f819d7502a814a5,d2eecf7d4aa6f0a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:37:07.158 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:37:07.158 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:37:07.158407838+08:00 2024-10-16 13:37:07.159 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:37:07.161756051+08:00 2024-10-16 13:37:07.159 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:37:07.158+08:00 2024-10-16 13:37:07.159 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:37:07.159+08:00 2024-10-16 13:37:07.160 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:07.160 INFO 4486 --- [host-check-agent-status2,9f819d7502a814a5,3e2fcd380e3d4c52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:37:07.160 INFO 4486 --- [host-check-agent-status1,9f819d7502a814a5,27612867c14c3d96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:37:07.161 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:37:07.161 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.161 INFO 4486 --- [host-check-agent-status3,9f819d7502a814a5,75a1ec8daeb5bd78] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:37:07.161 INFO 4486 --- [host-check-agent-status3,9f819d7502a814a5,75a1ec8daeb5bd78] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:37:07.161 INFO 4486 --- [host-check-agent-status3,9f819d7502a814a5,75a1ec8daeb5bd78] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:37:07.162 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:37:07.162 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:37:07.158+08:00 and status AVAILABLE 2024-10-16 13:37:07.162 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:37:07.159+08:00 and status AVAILABLE 2024-10-16 13:37:07.162 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:37:07.165120804+08:00 2024-10-16 13:37:07.163 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:37:07.165937125+08:00 2024-10-16 13:37:07.164 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:37:07.162+08:00 2024-10-16 13:37:07.164 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:37:07.163+08:00 2024-10-16 13:37:07.164 INFO 4486 --- [host-check-agent-status4,9f819d7502a814a5,d2eecf7d4aa6f0a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:37:07.164 INFO 4486 --- [host-check-agent-status4,9f819d7502a814a5,d2eecf7d4aa6f0a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:37:07.164 INFO 4486 --- [host-check-agent-status4,9f819d7502a814a5,d2eecf7d4aa6f0a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-agent-status2,9f819d7502a814a5,3e2fcd380e3d4c52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-agent-status1,9f819d7502a814a5,27612867c14c3d96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-agent-status1,9f819d7502a814a5,27612867c14c3d96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-agent-status2,9f819d7502a814a5,3e2fcd380e3d4c52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-agent-status1,9f819d7502a814a5,27612867c14c3d96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:37:07.165 INFO 4486 --- [host-check-agent-status2,9f819d7502a814a5,3e2fcd380e3d4c52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:37:07.166 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:37:07.167 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:37:07.163+08:00 and status AVAILABLE 2024-10-16 13:37:07.167 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:37:07.162+08:00 and status AVAILABLE 2024-10-16 13:37:07.170 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:37:07.170 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:37:07.173 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.174 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:37:07.175 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:07.175 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:07.176 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:07.177 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:07.179 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:07.179 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:07.180 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:07.181 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:07.184 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.185 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:37:07.194 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.195 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:37:07.197 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.198 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:37:07.198 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:07.203 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:07.203 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:07.204 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:07.207 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:07.208 INFO 4486 --- [ocp-schedules-23,edd6e33aec21ee59,4c4545b048237e63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:37:07.237 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:37:07.238 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:37:07.239 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,92203bdc35eb4502,28a31f23d41333a6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=92203bdc35eb4502, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:07.240 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,92203bdc35eb4502,28a31f23d41333a6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=92203bdc35eb4502, duration=1 ms] 2024-10-16 13:37:07.247 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:37:07.247 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:37:07.250 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:37:07.252 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:37:07.262 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:37:07.263 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:07.264 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:07.265 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:37:07.277 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:07.279 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:07.293 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:37:07.304 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:37:07.305 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:37:07.315 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:37:07.317 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:37:07.320 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:07.327 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:37:07.331 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:07.338 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:37:07.344 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:07.347 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:37:07.349 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:37:07.357 INFO 4486 --- [host-check-host-status3,a9a81f92651625a6,878316e297dcfa99] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:37:07.364 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:07.365 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:37:07.376 INFO 4486 --- [host-check-host-status4,a9a81f92651625a6,1f4727cc5d235a0a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:37:07.398 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:37:07.409 INFO 4486 --- [host-check-host-status2,a9a81f92651625a6,74b3d2343703eadf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:37:07.414 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:37:07.425 INFO 4486 --- [host-check-host-status1,a9a81f92651625a6,f374e8cb43429e15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:37:07.994 INFO 4486 --- [prometheus-minute-schedule-1,1dcdf2c9a4df3588,0c5ced20a012a011] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:07.999 INFO 4486 --- [prometheus-minute-schedule-1,1dcdf2c9a4df3588,0c5ced20a012a011] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:08.050 INFO 4486 --- [host-check-clockdiff4,1839eb7e19f1632c,005d7ca175ab3d61] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057028 3 3 , error= 2024-10-16 13:37:08.058 INFO 4486 --- [host-check-clockdiff2,1839eb7e19f1632c,b8fb3058382da60f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057028 3 3 , error= 2024-10-16 13:37:09.087 INFO 4486 --- [ocp-async-3,2d1c171ce018f5de,f899d0c7a63a5e6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:09.460 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,2243b14c40722585,85ce9a76cf6c3dc0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2243b14c40722585, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:09.460 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,2243b14c40722585,85ce9a76cf6c3dc0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2243b14c40722585, duration=1 ms] 2024-10-16 13:37:09.805 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:10.006 INFO 4486 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:37:10.006 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:37:10.024 INFO 4486 --- [Check obproxy1,a59973d5819acaf3,04c9001bc8ec7152] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:37:10.025 INFO 4486 --- [Check obproxy1,a59973d5819acaf3,04c9001bc8ec7152] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:10.025 INFO 4486 --- [Check obproxy2,1b8bdd1264b09f24,890aef23c888ebbf] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:37:10.025 INFO 4486 --- [Check obproxy2,1b8bdd1264b09f24,890aef23c888ebbf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:10.028 INFO 4486 --- [Collect obproxy1,4dcb6be8e3dda4ad,b0cd367f6c67cdee] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:37:10.029 INFO 4486 --- [Collect obproxy1,4dcb6be8e3dda4ad,b0cd367f6c67cdee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:10.029 INFO 4486 --- [Collect obproxy2,de8bd13257d0bf60,577e66191f1e2c70] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:37:10.029 INFO 4486 --- [Collect obproxy2,de8bd13257d0bf60,577e66191f1e2c70] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:10.048 INFO 4486 --- [Check obproxy2,1b8bdd1264b09f24,890aef23c888ebbf] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:37:10.048 INFO 4486 --- [Check obproxy2,1b8bdd1264b09f24,890aef23c888ebbf] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:37:10.052 INFO 4486 --- [Collect obproxy2,de8bd13257d0bf60,577e66191f1e2c70] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:37:10.053 INFO 4486 --- [Collect obproxy1,4dcb6be8e3dda4ad,b0cd367f6c67cdee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:37:10.056 INFO 4486 --- [Check obproxy1,a59973d5819acaf3,04c9001bc8ec7152] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:37:10.056 INFO 4486 --- [Check obproxy1,a59973d5819acaf3,04c9001bc8ec7152] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:37:10.058 INFO 4486 --- [Collect obproxy2,de8bd13257d0bf60,577e66191f1e2c70] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:37:10.058 INFO 4486 --- [Collect obproxy1,4dcb6be8e3dda4ad,b0cd367f6c67cdee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:37:10.061 INFO 4486 --- [Collect obproxy1,4dcb6be8e3dda4ad,b0cd367f6c67cdee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:37:10.063 INFO 4486 --- [Collect obproxy2,de8bd13257d0bf60,577e66191f1e2c70] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:37:10.428 INFO 4486 --- [ocp-schedules-14,10785cee5b20b7ea,61626be275af3527] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:37:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:37:10.994 INFO 4486 --- [prometheus-minute-schedule-1,8d560e57d7a37c17,8261d061ea96ba60] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:10.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d4b76d076eb5d9f4,c86554e02890c1bb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d4b76d076eb5d9f4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:10.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d4b76d076eb5d9f4,c86554e02890c1bb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d4b76d076eb5d9f4, duration=1 ms] 2024-10-16 13:37:11.001 INFO 4486 --- [prometheus-minute-schedule-1,8d560e57d7a37c17,8261d061ea96ba60] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:37:11.088 INFO 4486 --- [ocp-async-6,3522f1d209b88f14,3bf844e755156c3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:11.351 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c00e20e986559f6a,a3cf98ff5f29165f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c00e20e986559f6a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:11.351 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c00e20e986559f6a,a3cf98ff5f29165f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c00e20e986559f6a, duration=1 ms] 2024-10-16 13:37:12.805 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:12.993 INFO 4486 --- [ocp-async-10,0787c0e54916d00d,98189aa0e8579a69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:37:13.088 INFO 4486 --- [ocp-async-9,3ab50102670e1d25,369d9aa6884a6614] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:13.360 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056960, nodeCount=1000, affectRows=1000 2024-10-16 13:37:13.364 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056960, nodeCount=1000, validCount=1000 2024-10-16 13:37:13.369 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056960, nodeCount=1000, affectRows=1000 2024-10-16 13:37:13.374 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056960, nodeCount=1000, validCount=1000 2024-10-16 13:37:13.377 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056960, nodeCount=1000, affectRows=1000 2024-10-16 13:37:13.381 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056960, nodeCount=998, validCount=998 2024-10-16 13:37:13.385 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729056960, nodeCount=507, affectRows=507 2024-10-16 13:37:13.387 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729056960, nodeCount=506, validCount=506 2024-10-16 13:37:13.468 INFO 4486 --- [ocp-schedules-29,c732a302f47f1383,893903fafd2fbac7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:37:13.516 WARN 4486 --- [scanner-13258853-1,c732a302f47f1383,93b741cd06cd77a7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:37:13.604 INFO 4486 --- [alarm-detect-15,c732a302f47f1383,a1367d63b7365eb7] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057005 ~ 1729057023 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057020, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057020, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057020=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:37:13.994 INFO 4486 --- [prometheus-minute-schedule-1,e9e8cf5662992f24,a09f2b5dfd8d8657] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:13.999 INFO 4486 --- [prometheus-minute-schedule-1,e9e8cf5662992f24,a09f2b5dfd8d8657] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:14.478 INFO 4486 --- [metric-archive-async-28,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056960, nodeCount=184, affectRows=184 2024-10-16 13:37:14.479 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056960, nodeCount=167, affectRows=167 2024-10-16 13:37:14.481 INFO 4486 --- [metric-archive-async-11,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729056960, nodeCount=129, affectRows=129 2024-10-16 13:37:14.688 INFO 4486 --- [metric-archive-async-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056960, nodeCount=129, validCount=129 2024-10-16 13:37:14.757 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056960, nodeCount=167, validCount=167 2024-10-16 13:37:14.760 INFO 4486 --- [metric-archive-async-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729056960, nodeCount=184, validCount=184 2024-10-16 13:37:15.088 INFO 4486 --- [ocp-async-7,2d5a325662453341,cef034165d92059c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:15.806 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:16.994 INFO 4486 --- [prometheus-minute-schedule-1,88f5fbcecbb27151,37dbf77602a9f173] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:17.001 INFO 4486 --- [prometheus-minute-schedule-1,88f5fbcecbb27151,37dbf77602a9f173] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:17.070 INFO 4486 --- [ocp-schedules-22,6282368eb0569354,267d3b631d4518d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:37:17.089 INFO 4486 --- [ocp-async-2,74d95275ba88c6cf,bc892d279278d912] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:17.110 INFO 4486 --- [ocp-schedules-4,c4c85223f586ae2e,0be3b9ee3a57f27c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:37:17.159 INFO 4486 --- [ocp-schedules-1,864b5aca05972f62,d9696cae09703fe7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:37:17.245 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ada01f1e12da3aeb,98977f4680509084] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ada01f1e12da3aeb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:17.245 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ada01f1e12da3aeb,98977f4680509084] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ada01f1e12da3aeb, duration=2 ms] 2024-10-16 13:37:18.807 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:18.911 INFO 4486 --- [ocp-schedules-19,d63a1844e335b1fb,4e737fdb7e90f2c5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:37:19.089 INFO 4486 --- [ocp-async-5,48c2e8e3a3872f5b,d45e7a386ff02e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:19.464 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,28773617ed67bc41,7597ce0686091379] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=28773617ed67bc41, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:19.464 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,28773617ed67bc41,7597ce0686091379] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=28773617ed67bc41, duration=1 ms] 2024-10-16 13:37:19.994 INFO 4486 --- [prometheus-minute-schedule-1,e1f61725b18cb4fc,37c7cdb586137982] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:19.999 INFO 4486 --- [prometheus-minute-schedule-1,e1f61725b18cb4fc,37c7cdb586137982] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:20.433 INFO 4486 --- [ocp-schedules-24,5e8fbe8a7373a61e,1a2970c16dc51bbb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:37:20.868 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 13:37:20.868 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 13:37:20.996 INFO 4486 --- [prometheus-cache-monitor1,926a19678b13b448,ca0280d38ae28a98] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3452,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:37:20.996 INFO 4486 --- [prometheus-cache-monitor1,926a19678b13b448,ca0280d38ae28a98] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:37:21.001 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,a9f6a0527f593e0d,96b38a3f65576457] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a9f6a0527f593e0d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:21.001 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,a9f6a0527f593e0d,96b38a3f65576457] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a9f6a0527f593e0d, duration=1 ms] 2024-10-16 13:37:21.089 INFO 4486 --- [ocp-async-9,a3336063ead04c6a,60f91667aa9edad7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:21.355 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,37458c8520f32e67,aaa9616f52c1f1bd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=37458c8520f32e67, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:21.355 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,37458c8520f32e67,aaa9616f52c1f1bd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=37458c8520f32e67, duration=1 ms] 2024-10-16 13:37:21.808 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:22.873 INFO 4486 --- [ocp-schedules-4,62c778d93688666a,e24b8724c5df2182] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:37:22.994 INFO 4486 --- [ocp-async-3,7a2b74f90fb4316b,44a85e2b1c705cd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:37:22.994 INFO 4486 --- [prometheus-minute-schedule-1,adf57c44d5866988,10d607b1b6d18f4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:23.001 INFO 4486 --- [prometheus-minute-schedule-1,adf57c44d5866988,10d607b1b6d18f4f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:23.089 INFO 4486 --- [ocp-async-1,34126212685775e5,1b00bf935f6ca095] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:23.683 INFO 4486 --- [ocp-schedules-13,37090ad4389b693d,1934375dd90784eb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:37:23.709 WARN 4486 --- [scanner-13259157-1,37090ad4389b693d,3d537511e64d2001] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:37:24.353 INFO 4486 --- [ocp-schedules-6,67a9b68d3d3bbc0e,9c227a2e9f064542] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:37:24.396 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,24a760d5566229b7,e1d6de2f11042b0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=24a760d5566229b7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:37:24.397 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,24a760d5566229b7,e1d6de2f11042b0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=24a760d5566229b7, duration=1 ms] 2024-10-16 13:37:24.724 INFO 4486 --- [ocp-schedules-10,017860f1fb9e9cba,55ba5f61b5387b3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:37:24.750 INFO 4486 --- [ocp-schedules-20,ee8f72547ae353fa,a1c8e6e242b0a13f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:37:24.750 INFO 4486 --- [ocp-schedules-20,ee8f72547ae353fa,a1c8e6e242b0a13f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:37:24.785 INFO 4486 --- [Collect all tenants major compactions1,ee8f72547ae353fa,d2a64f6800155fe3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:37:24.785 INFO 4486 --- [Collect all tenants major compactions3,ee8f72547ae353fa,8c34f91b8fe669d9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:37:24.785 INFO 4486 --- [Collect all tenants major compactions2,ee8f72547ae353fa,58e56cb20b4f9fc7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:37:24.786 INFO 4486 --- [Collect all tenants major compactions5,ee8f72547ae353fa,dc7a4e7fb1060c12] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:37:24.786 INFO 4486 --- [Collect all tenants major compactions4,ee8f72547ae353fa,085ff9e43a028921] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:37:24.797 INFO 4486 --- [Collect all tenants major compactions2,ee8f72547ae353fa,58e56cb20b4f9fc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:24.798 INFO 4486 --- [Collect all tenants major compactions1,ee8f72547ae353fa,d2a64f6800155fe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:24.799 INFO 4486 --- [Collect all tenants major compactions2,ee8f72547ae353fa,58e56cb20b4f9fc7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:37:24.800 INFO 4486 --- [Collect all tenants major compactions1,ee8f72547ae353fa,d2a64f6800155fe3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:37:24.803 INFO 4486 --- [Collect all tenants major compactions1,ee8f72547ae353fa,d2a64f6800155fe3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:37:24.803 INFO 4486 --- [Collect all tenants major compactions2,ee8f72547ae353fa,58e56cb20b4f9fc7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:37:24.803 INFO 4486 --- [Collect all tenants major compactions1,ee8f72547ae353fa,d2a64f6800155fe3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:37:24.803 INFO 4486 --- [Collect all tenants major compactions2,ee8f72547ae353fa,58e56cb20b4f9fc7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:37:24.805 INFO 4486 --- [Collect all tenants major compactions3,ee8f72547ae353fa,8c34f91b8fe669d9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:24.807 INFO 4486 --- [Collect all tenants major compactions3,ee8f72547ae353fa,8c34f91b8fe669d9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:37:24.809 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:24.812 INFO 4486 --- [Collect all tenants major compactions3,ee8f72547ae353fa,8c34f91b8fe669d9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:37:24.812 INFO 4486 --- [Collect all tenants major compactions3,ee8f72547ae353fa,8c34f91b8fe669d9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:37:24.821 INFO 4486 --- [Collect all tenants major compactions5,ee8f72547ae353fa,dc7a4e7fb1060c12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:24.821 INFO 4486 --- [Collect all tenants major compactions4,ee8f72547ae353fa,085ff9e43a028921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:24.822 INFO 4486 --- [Collect all tenants major compactions5,ee8f72547ae353fa,dc7a4e7fb1060c12] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:37:24.823 INFO 4486 --- [Collect all tenants major compactions4,ee8f72547ae353fa,085ff9e43a028921] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:37:24.827 INFO 4486 --- [Collect all tenants major compactions4,ee8f72547ae353fa,085ff9e43a028921] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:37:24.827 INFO 4486 --- [Collect all tenants major compactions4,ee8f72547ae353fa,085ff9e43a028921] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:37:24.827 INFO 4486 --- [Collect all tenants major compactions5,ee8f72547ae353fa,dc7a4e7fb1060c12] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:37:24.828 INFO 4486 --- [Collect all tenants major compactions5,ee8f72547ae353fa,dc7a4e7fb1060c12] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:37:24.829 INFO 4486 --- [ocp-schedules-20,ee8f72547ae353fa,a1c8e6e242b0a13f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:37:25.090 INFO 4486 --- [ocp-async-10,999a6ba9cd55e11e,c5ca12534e4a240e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:25.994 INFO 4486 --- [prometheus-minute-schedule-1,bfa2ed9441dc42a2,e2c06aeb92266d03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:26.001 INFO 4486 --- [prometheus-minute-schedule-1,bfa2ed9441dc42a2,e2c06aeb92266d03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:27.008 INFO 4486 --- [ocp-schedules-30,fb3aecff77656aa6,ed277256c0a48f18] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:37:27.008+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:37:27.010 INFO 4486 --- [ocp-schedules-30,fb3aecff77656aa6,ed277256c0a48f18] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:37:27.011 INFO 4486 --- [ocp-schedules-30,fb3aecff77656aa6,ed277256c0a48f18] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:37:27.012 INFO 4486 --- [ocp-schedules-30,fb3aecff77656aa6,ed277256c0a48f18] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:37:27.090 INFO 4486 --- [ocp-async-9,c1105946e95c7ccf,9a87339943b8b39c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:27.253 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1ad758c5d693f669,0e99821ef847cf39] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1ad758c5d693f669, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:27.253 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1ad758c5d693f669,0e99821ef847cf39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1ad758c5d693f669, duration=2 ms] 2024-10-16 13:37:27.810 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:28.994 INFO 4486 --- [prometheus-minute-schedule-1,96021f80c3c845e1,78c412c991fd4c73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:29.000 INFO 4486 --- [prometheus-minute-schedule-1,96021f80c3c845e1,78c412c991fd4c73] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:29.090 INFO 4486 --- [ocp-async-6,3fbf10eaa8a3000b,c544ea9ddd5b605e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:29.468 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,9a63945fbe51b56a,1312690e5219164b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9a63945fbe51b56a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:29.469 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,9a63945fbe51b56a,1312690e5219164b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9a63945fbe51b56a, duration=1 ms] 2024-10-16 13:37:30.103 INFO 4486 --- [prometheus-second-schedule-1,9b0a8f0c1c0f75c5,6765075c9895c1d9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:37:30.439 INFO 4486 --- [ocp-schedules-3,6d3a6d13ecaf9267,a56def36d5741fa6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:37:30.811 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:31.037 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,6cfa0f1e6f30b40d,cb45d4890472eb33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6cfa0f1e6f30b40d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:31.037 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,6cfa0f1e6f30b40d,cb45d4890472eb33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6cfa0f1e6f30b40d, duration=1 ms] 2024-10-16 13:37:31.090 INFO 4486 --- [ocp-async-5,eda6275892a8fea6,344352a1e265ce90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:31.361 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,58cbf7ba069cb623,59ca536e40f71375] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=58cbf7ba069cb623, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:31.361 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,58cbf7ba069cb623,59ca536e40f71375] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=58cbf7ba069cb623, duration=2 ms] 2024-10-16 13:37:31.994 INFO 4486 --- [prometheus-minute-schedule-1,c415a410e9b4995e,3c558ebe6223af24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:32.000 INFO 4486 --- [prometheus-minute-schedule-1,c415a410e9b4995e,3c558ebe6223af24] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:32.994 INFO 4486 --- [ocp-async-10,f98812134c236966,e2e66a24f34f93d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:37:33.090 INFO 4486 --- [ocp-async-9,1bf34eb95ea20890,0b86509fe2e11daa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:33.812 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:33.864 INFO 4486 --- [ocp-schedules-8,aa6df2a3f074c65e,5018fd06a9dc3c6b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:37:33.905 WARN 4486 --- [scanner-13259390-1,aa6df2a3f074c65e,ed0ab9e011f18697] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:37:34.001 INFO 4486 --- [alarm-detect-9,aa6df2a3f074c65e,88426781cac59df9] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057020 ~ 1729057043 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057035, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057035, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057035=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:37:34.994 INFO 4486 --- [prometheus-minute-schedule-1,c7c9659765cf2321,efd975ee36b789a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:35.001 INFO 4486 --- [prometheus-minute-schedule-1,c7c9659765cf2321,efd975ee36b789a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:35.091 INFO 4486 --- [ocp-async-3,91a148631dcbff81,7d1df962de218940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:36.033 INFO 4486 --- [ocp-schedules-16,7c21456fcca1dcbf,f625ea1fe718dd77] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:37:36.034 INFO 4486 --- [ocp-schedules-16,c5ed3c009139f197,b15b748251c273af] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:37:36.036 INFO 4486 --- [ocp-schedules-16,c5ed3c009139f197,b15b748251c273af] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:36.064 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.065 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:36.076 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.077 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:36.101 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.102 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.102 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:37:36.103 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.104 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:36.133 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.134 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.134 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:36.135 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:37:36.135 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:37:36.140 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:36.150 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.150 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:36.160 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.161 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,03acb2b0c8ac8b8f,a46762166784a925] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=03acb2b0c8ac8b8f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:37:36.162 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,03acb2b0c8ac8b8f,a46762166784a925] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=03acb2b0c8ac8b8f, duration=2 ms] 2024-10-16 13:37:36.168 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.169 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.169 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:36.170 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:37:36.175 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:37:36.179 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:36.181 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:36.202 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:37:36.219 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.220 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.221 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:37:36.232 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:37:36.233 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:37:36.269 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.271 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.272 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:37:36.276 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.279 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:37:36.281 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:37:36.282 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:36.291 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.292 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.293 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:37:36.302 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:37:36.302 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:37:36.309 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.309 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:36.335 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.335 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:36.360 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.360 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:36.387 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.387 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:37:36.418 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.418 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:37:36.446 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:37:36.472 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.474 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.475 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:36.479 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:37:36.484 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:37:36.488 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:37:36.490 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:37:36.513 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:37:36.646 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:37:36.697 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:36.699 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:36.700 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:37:36.727 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:37:36.727 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:37:36.813 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:37.063 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:37:37.064 INFO 4486 --- [ocp-schedules-28,8ce681fc9606b694,edbaf0878e81a3f4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:37:37.066 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:37:37.066 INFO 4486 --- [ocp-schedules-4,99cd8650deb263c7,029948695e294727] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:37:37.068 INFO 4486 --- [ocp-schedules-4,99cd8650deb263c7,029948695e294727] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:37.071 INFO 4486 --- [ocp-schedules-28,8ce681fc9606b694,edbaf0878e81a3f4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:37:37.072 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:37:37.081 INFO 4486 --- [ocp-schedules-2,40d8c2b2a3c2f1b2,34861d583ca4e223] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:37:37.083 INFO 4486 --- [ocp-schedules-2,40d8c2b2a3c2f1b2,34861d583ca4e223] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:37:37.090 INFO 4486 --- [ocp-schedules-21,cc207a203a67d2de,0aa7f10336b9da32] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:37:37.090 INFO 4486 --- [ocp-schedules-3,e0167d18fe933553,6596e81e99c2f87a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:37:37.091 INFO 4486 --- [ocp-schedules-27,0b39f4d75dd3f97a,6687cd29746d4833] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:37:37.092 INFO 4486 --- [ocp-async-5,59782970545192c0,43d4d311c4f54766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:37.092 INFO 4486 --- [host-check-clockdiff3,cc207a203a67d2de,f29277c441401cf8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:37:37.092 INFO 4486 --- [host-check-clockdiff4,cc207a203a67d2de,365f89f0d17f40f8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:37:37.092 INFO 4486 --- [host-check-clockdiff2,cc207a203a67d2de,ffa1e349208620f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:37:37.092 INFO 4486 --- [host-check-clockdiff1,cc207a203a67d2de,8c8f0fd883452c5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:37:37.092 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:37:37.092 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:37:37.093 INFO 4486 --- [host-check-agent-status1,0b39f4d75dd3f97a,0568b3011d2fd01c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:37:37.094 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:37:37.096 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:37:37.100 INFO 4486 --- [host-check-agent-status3,0b39f4d75dd3f97a,ce221457b9535032] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:37:37.100 INFO 4486 --- [host-check-agent-status2,0b39f4d75dd3f97a,d8de6a8103b1a533] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:37:37.101 INFO 4486 --- [host-check-agent-status4,0b39f4d75dd3f97a,d6f5434b183b3819] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:37:37.101 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:37.103 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:37.104 INFO 4486 --- [host-check-agent-status1,0b39f4d75dd3f97a,0568b3011d2fd01c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:37.104 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:37.104 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:37.104 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:37.105 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:37.105 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:37.106 INFO 4486 --- [host-check-agent-status1,0b39f4d75dd3f97a,0568b3011d2fd01c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:37.107 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.107 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:37.108 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:37.129 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:37:37.134 INFO 4486 --- [host-check-agent-status4,0b39f4d75dd3f97a,d6f5434b183b3819] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:37.134 INFO 4486 --- [host-check-clockdiff1,cc207a203a67d2de,8c8f0fd883452c5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057057 0 0 , error= 2024-10-16 13:37:37.142 INFO 4486 --- [host-check-agent-status4,0b39f4d75dd3f97a,d6f5434b183b3819] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:37.142 INFO 4486 --- [host-check-agent-status3,0b39f4d75dd3f97a,ce221457b9535032] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:37.142 INFO 4486 --- [host-check-clockdiff3,cc207a203a67d2de,f29277c441401cf8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057057 3 3 , error= 2024-10-16 13:37:37.146 INFO 4486 --- [host-check-agent-status2,0b39f4d75dd3f97a,d8de6a8103b1a533] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:37.148 INFO 4486 --- [host-check-agent-status3,0b39f4d75dd3f97a,ce221457b9535032] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:37.148 INFO 4486 --- [host-check-agent-status2,0b39f4d75dd3f97a,d8de6a8103b1a533] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:37.166 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.167 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:37:37.169 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:37:37.169 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:37:37.170 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:37.173 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:37:37.173 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:37:37.174 INFO 4486 --- [host-check-agent-status1,0b39f4d75dd3f97a,0568b3011d2fd01c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:37:37.174 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:37:37.177339503+08:00 2024-10-16 13:37:37.175 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:37:37.174615490+08:00 2024-10-16 13:37:37.175 INFO 4486 --- [host-check-agent-status3,0b39f4d75dd3f97a,ce221457b9535032] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:37:37.180 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.180 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:37:37.174+08:00 2024-10-16 13:37:37.181 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:37:37.178748989+08:00 2024-10-16 13:37:37.181 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:37:37.175+08:00 2024-10-16 13:37:37.184 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:37:37.184 INFO 4486 --- [host-check-agent-status1,0b39f4d75dd3f97a,0568b3011d2fd01c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:37:37.184 INFO 4486 --- [host-check-agent-status1,0b39f4d75dd3f97a,0568b3011d2fd01c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:37:37.184 INFO 4486 --- [host-check-agent-status1,0b39f4d75dd3f97a,0568b3011d2fd01c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:37:37.185 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:37:37.181+08:00 2024-10-16 13:37:37.187 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:37:37.189710971+08:00 2024-10-16 13:37:37.187 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:37:37.175+08:00 and status AVAILABLE 2024-10-16 13:37:37.187 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:37:37.174+08:00 and status AVAILABLE 2024-10-16 13:37:37.187 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:37:37.187+08:00 2024-10-16 13:37:37.188 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:37:37.188 INFO 4486 --- [host-check-agent-status3,0b39f4d75dd3f97a,ce221457b9535032] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:37:37.188 INFO 4486 --- [host-check-agent-status3,0b39f4d75dd3f97a,ce221457b9535032] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:37:37.188 INFO 4486 --- [host-check-agent-status3,0b39f4d75dd3f97a,ce221457b9535032] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:37:37.188 INFO 4486 --- [host-check-agent-status2,0b39f4d75dd3f97a,d8de6a8103b1a533] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:37:37.188 INFO 4486 --- [host-check-agent-status4,0b39f4d75dd3f97a,d6f5434b183b3819] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:37:37.188 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:37:37.181+08:00 and status AVAILABLE 2024-10-16 13:37:37.190 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:37:37.190 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:37:37.187+08:00 and status AVAILABLE 2024-10-16 13:37:37.190 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:37:37.191 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:37:37.191 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.192 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:37:37.192 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:37:37.192 INFO 4486 --- [host-check-agent-status4,0b39f4d75dd3f97a,d6f5434b183b3819] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:37:37.192 INFO 4486 --- [host-check-agent-status4,0b39f4d75dd3f97a,d6f5434b183b3819] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:37:37.192 INFO 4486 --- [host-check-agent-status4,0b39f4d75dd3f97a,d6f5434b183b3819] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:37:37.193 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.193 INFO 4486 --- [host-check-agent-status2,0b39f4d75dd3f97a,d8de6a8103b1a533] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:37:37.193 INFO 4486 --- [host-check-agent-status2,0b39f4d75dd3f97a,d8de6a8103b1a533] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:37:37.193 INFO 4486 --- [host-check-agent-status2,0b39f4d75dd3f97a,d8de6a8103b1a533] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:37:37.194 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.194 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:37:37.196 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.196 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:37:37.198 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:37.199 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:37.199 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:37.200 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:37.200 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:37.200 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:37.201 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:37.201 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:37.204 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.204 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.204 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.204 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.204 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.205 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.205 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.206 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:37:37.220 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.220 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:37:37.223 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.223 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:37:37.224 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:37:37.225 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:37.226 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:37.226 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:37.227 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:37:37.233 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.234 INFO 4486 --- [ocp-schedules-11,e2384eda4ed29f59,0bf1b96caa52cce3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:37:37.254 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:37:37.257 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4acb29770e2b2c5a,f1b7dccded0a1c8e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4acb29770e2b2c5a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:37.258 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4acb29770e2b2c5a,f1b7dccded0a1c8e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4acb29770e2b2c5a, duration=1 ms] 2024-10-16 13:37:37.262 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:37:37.264 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:37:37.270 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:37:37.274 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:37:37.282 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:37.282 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:37:37.282 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:37:37.293 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:37.300 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:37.311 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:37:37.312 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:37:37.320 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:37:37.328 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:37:37.334 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:37:37.336 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:37.344 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:37:37.344 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:37:37.354 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:37:37.361 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:37.364 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:37:37.371 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:37.373 INFO 4486 --- [host-check-host-status3,e0167d18fe933553,c750233e6216b069] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:37:37.382 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:37:37.392 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:37:37.396 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:37:37.409 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:37:37.409 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:37:37.410 INFO 4486 --- [host-check-host-status4,e0167d18fe933553,c4fad314a44f7fc7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:37:37.421 INFO 4486 --- [host-check-host-status2,e0167d18fe933553,41b27f918389b44c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:37:37.449 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:37.451 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.452 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:37:37.453 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.454 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.455 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:37:37.469 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.469 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.469 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.469 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.469 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.469 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.470 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:37:37.479 INFO 4486 --- [host-check-host-status1,e0167d18fe933553,dd328ae0bb9800c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:37:37.496 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:37:37.532 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:37:37.575 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:37.590 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:37.591 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.593 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:37:37.594 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:37:37.595 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.595 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:37:37.596 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.597 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.598 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:37:37.606 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.606 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.606 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.606 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.606 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.607 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.615 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:37:37.823 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:37:37.868 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:37.870 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.872 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:37:37.873 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.874 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.876 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:37:37.887 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.887 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.887 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.887 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.887 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:37.888 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:37.906 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:37:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:37:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:37:37.994 INFO 4486 --- [prometheus-minute-schedule-1,883ab86bdd073063,973ca086e210a848] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:38.000 INFO 4486 --- [prometheus-minute-schedule-1,883ab86bdd073063,973ca086e210a848] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:38.058 INFO 4486 --- [host-check-clockdiff4,cc207a203a67d2de,365f89f0d17f40f8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057058 3 3 , error= 2024-10-16 13:37:38.116 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:37:38.157 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:37:38.159 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:38.161 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:37:38.161 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:38.162 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:38.163 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:37:38.173 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:38.173 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:38.173 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:38.173 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:38.173 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:37:38.174 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:37:38.239 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:37:38.428 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:37:38.242415, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:37:38.429 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:37:38.242415, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:37:38.466 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:37:38.242415} } 2024-10-16 13:37:38.467 INFO 4486 --- [ocp-schedules-19,811a2579a4044159,73e2db37efadce53] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3452,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:37:38.468 INFO 4486 --- [ocp-schedules-19,811a2579a4044159,73e2db37efadce53] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:37:38.529 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:37:38.537 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:37:38.540 INFO 4486 --- [ocp-schedules-26,4996a7a8f1093c90,5fa13311c5af6b48] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:37:38.663 INFO 4486 --- [ocp-schedules-14,6f6fe060df017a0d,f2e282ad9f354e52] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:37:38.665 INFO 4486 --- [ocp-schedules-14,6f6fe060df017a0d,f2e282ad9f354e52] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:38.913 INFO 4486 --- [host-check-clockdiff2,cc207a203a67d2de,ffa1e349208620f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057058 3 3 , error= 2024-10-16 13:37:38.915 INFO 4486 --- [ocp-schedules-1,adf5d3dc5e0ac752,de2e012e4f9eb821] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:37:39.092 INFO 4486 --- [ocp-async-9,91f46fc1a67ed8ff,82f8564660a097e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:39.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,decad10dc4c8d276,b3696054845b7604] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=decad10dc4c8d276, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:39.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,decad10dc4c8d276,b3696054845b7604] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=decad10dc4c8d276, duration=1 ms] 2024-10-16 13:37:39.814 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:40.006 INFO 4486 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:37:40.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:37:40.024 INFO 4486 --- [Check obproxy2,c05df531b7cc185a,3be487c59e081121] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:37:40.025 INFO 4486 --- [Check obproxy2,c05df531b7cc185a,3be487c59e081121] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:40.026 INFO 4486 --- [Check obproxy1,c100d84cf37e235c,27cadbf03d7755a0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:37:40.026 INFO 4486 --- [Check obproxy1,c100d84cf37e235c,27cadbf03d7755a0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:40.030 INFO 4486 --- [Collect obproxy2,e475310a7e123128,a91534f600d58fc5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:37:40.030 INFO 4486 --- [Collect obproxy1,c1db7eb51d9aa2b2,fe6ab77dc1ad5c89] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:37:40.030 INFO 4486 --- [Collect obproxy2,e475310a7e123128,a91534f600d58fc5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:40.030 INFO 4486 --- [Collect obproxy1,c1db7eb51d9aa2b2,fe6ab77dc1ad5c89] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:37:40.051 INFO 4486 --- [Check obproxy2,c05df531b7cc185a,3be487c59e081121] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:37:40.051 INFO 4486 --- [Check obproxy2,c05df531b7cc185a,3be487c59e081121] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:37:40.056 INFO 4486 --- [Collect obproxy1,c1db7eb51d9aa2b2,fe6ab77dc1ad5c89] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:37:40.062 INFO 4486 --- [Collect obproxy1,c1db7eb51d9aa2b2,fe6ab77dc1ad5c89] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:37:40.063 INFO 4486 --- [Check obproxy1,c100d84cf37e235c,27cadbf03d7755a0] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:37:40.063 INFO 4486 --- [Check obproxy1,c100d84cf37e235c,27cadbf03d7755a0] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:37:40.065 INFO 4486 --- [Collect obproxy1,c1db7eb51d9aa2b2,fe6ab77dc1ad5c89] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:37:40.069 INFO 4486 --- [Collect obproxy2,e475310a7e123128,a91534f600d58fc5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:37:40.075 INFO 4486 --- [Collect obproxy2,e475310a7e123128,a91534f600d58fc5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:37:40.079 INFO 4486 --- [Collect obproxy2,e475310a7e123128,a91534f600d58fc5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:37:40.444 INFO 4486 --- [ocp-schedules-15,6d68941a2e77910f,1bb4f840d0de10fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:37:40.994 INFO 4486 --- [prometheus-minute-schedule-1,b343efd225e3990f,e801d78977031725] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:40.995 INFO 4486 --- [prometheus-hour-schedule-1,1001711ebe31d40c,4a478a6ebd525e96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:37:41.001 INFO 4486 --- [prometheus-minute-schedule-1,b343efd225e3990f,e801d78977031725] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:41.043 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9e7b7096c14af34f,d315b2a428ea32e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9e7b7096c14af34f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:41.043 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9e7b7096c14af34f,d315b2a428ea32e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9e7b7096c14af34f, duration=1 ms] 2024-10-16 13:37:41.092 INFO 4486 --- [ocp-async-6,220bcd0e7de10470,ccc4c11292138a76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:41.365 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,db716e2037de2d87,e20623a81c39149d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=db716e2037de2d87, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:41.365 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,db716e2037de2d87,e20623a81c39149d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=db716e2037de2d87, duration=1 ms] 2024-10-16 13:37:42.096 INFO 4486 --- [ocp-schedules-13,1caca20e55909e20,a78c41b0271fd940] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:37:42.814 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:42.994 INFO 4486 --- [ocp-async-1,b636b515e95ece8c,a0524734ad8e942e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:37:43.092 INFO 4486 --- [ocp-async-8,201c66ca2b0cd4c2,c3253fc79ac2cebf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:43.994 INFO 4486 --- [prometheus-minute-schedule-1,7c22ccf6a6077a8f,5a77426ea1c68854] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:44.001 INFO 4486 --- [prometheus-minute-schedule-1,7c22ccf6a6077a8f,5a77426ea1c68854] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:44.137 INFO 4486 --- [ocp-schedules-28,373b3273e0b473ce,3b9252826ffd3cef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:37:44.167 WARN 4486 --- [scanner-13259750-1,373b3273e0b473ce,2ead36bcac8440a5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:37:44.217 INFO 4486 --- [alarm-detect-17,373b3273e0b473ce,791c6176f5926efc] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057035 ~ 1729057054 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057050, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057050, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057050=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:37:45.092 INFO 4486 --- [ocp-async-4,c3839aa43366f89f,98667bcc19c8a152] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:45.465 INFO 4486 --- [ocp-schedules-8,8648e0b6fe6762fb,ee97b13ed8a38efb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:37:45.471 INFO 4486 --- [ocp-schedules-8,8648e0b6fe6762fb,ee97b13ed8a38efb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:37:45.471 INFO 4486 --- [ocp-schedules-8,8648e0b6fe6762fb,ee97b13ed8a38efb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:37:45.485 INFO 4486 --- [ocp-schedules-22,8424b0294ad1aef9,803a42bae2b8ce22] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:37:45.485 INFO 4486 --- [ocp-schedules-22,8424b0294ad1aef9,803a42bae2b8ce22] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:37:45.816 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:46.994 INFO 4486 --- [prometheus-minute-schedule-1,95d87aaa722b34d5,f3047969e1fcf9a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:47.001 INFO 4486 --- [prometheus-minute-schedule-1,95d87aaa722b34d5,f3047969e1fcf9a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:47.071 INFO 4486 --- [ocp-schedules-4,99cd8650deb263c7,029948695e294727] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:37:47.093 INFO 4486 --- [ocp-async-3,9fedf3fb84237250,9bdcfce796951071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:47.113 INFO 4486 --- [ocp-schedules-11,0e080ea850129145,edc9b33da463e3f5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:37:47.174 INFO 4486 --- [ocp-schedules-2,40d8c2b2a3c2f1b2,34861d583ca4e223] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:37:47.265 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bda6c8428b038b6e,9065c46f925e2a75] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bda6c8428b038b6e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:47.266 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bda6c8428b038b6e,9065c46f925e2a75] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bda6c8428b038b6e, duration=3 ms] 2024-10-16 13:37:48.694 INFO 4486 --- [ocp-schedules-14,6f6fe060df017a0d,f2e282ad9f354e52] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:37:48.758 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:37:48.817 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:49.093 INFO 4486 --- [ocp-async-8,a61b9f2e8cd3ea23,881d12921c3eed0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:49.478 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2ad0d25c4fd49ecb,216202ca124bfeed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2ad0d25c4fd49ecb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:49.478 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2ad0d25c4fd49ecb,216202ca124bfeed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2ad0d25c4fd49ecb, duration=1 ms] 2024-10-16 13:37:49.994 INFO 4486 --- [prometheus-minute-schedule-1,c1961e577bf5f384,38aeef95333c327f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:50.000 INFO 4486 --- [prometheus-minute-schedule-1,c1961e577bf5f384,38aeef95333c327f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:50.193 INFO 4486 --- [ocp-schedules-10,b60930fa8b047d1e,972e7892649a2c15] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:37:50.193 INFO 4486 --- [ocp-schedules-10,b60930fa8b047d1e,972e7892649a2c15] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:37:50.193 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:37:50.193 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:37:50.193 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:37:50.193 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:37:50.193 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:37:50.199 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:37:50.199 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:50.199 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:50.199 INFO 4486 --- [ocp-schedules-10,b60930fa8b047d1e,972e7892649a2c15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:37:50.199 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:50.200 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:50.200 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:37:50.200 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:50.201 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:50.201 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:50.201 INFO 4486 --- [ocp-schedules-10,b60930fa8b047d1e,972e7892649a2c15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:37:50.203 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:50.225 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:37:50.225 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:37:50.226 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:37:50.226 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:37:50.230 INFO 4486 --- [ocp-schedules-10,b60930fa8b047d1e,972e7892649a2c15] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:37:50.231 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:37:50.234 INFO 4486 --- [ocp-schedules-10,b60930fa8b047d1e,972e7892649a2c15] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:37:50.398 INFO 4486 --- [ocp-schedules-20,6ab0d800200ce026,c1a1ea2cabc09379] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:37:50.398 INFO 4486 --- [ocp-schedules-20,6ab0d800200ce026,c1a1ea2cabc09379] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:37:50.399 INFO 4486 --- [ocp-schedules-20,6ab0d800200ce026,c1a1ea2cabc09379] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:37:50.399 INFO 4486 --- [ocp-schedules-20,6ab0d800200ce026,c1a1ea2cabc09379] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:37:50.400 INFO 4486 --- [ocp-schedules-20,6ab0d800200ce026,c1a1ea2cabc09379] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:37:50.400 INFO 4486 --- [ocp-schedules-20,6ab0d800200ce026,c1a1ea2cabc09379] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:37:50.449 INFO 4486 --- [ocp-schedules-5,9eb44bbfc7cce163,b67e453f3031f88a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:37:51.047 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,47e0afa576b6d1d9,02d8bd32416e1461] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=47e0afa576b6d1d9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:51.047 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,47e0afa576b6d1d9,02d8bd32416e1461] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=47e0afa576b6d1d9, duration=1 ms] 2024-10-16 13:37:51.093 INFO 4486 --- [ocp-async-4,71557cbde7c1b949,a0e045568253ac5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:51.370 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,468a56a69f16d93b,5d9744ae0bb5bb2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=468a56a69f16d93b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:51.370 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,468a56a69f16d93b,5d9744ae0bb5bb2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=468a56a69f16d93b, duration=1 ms] 2024-10-16 13:37:51.818 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:52.994 INFO 4486 --- [prometheus-minute-schedule-1,4c40f1f1c0a2e115,83dc5b6747b40cc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:52.994 INFO 4486 --- [ocp-async-4,72cc31c8fc45e9f2,9065d82dfa50b920] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:37:53.001 INFO 4486 --- [prometheus-minute-schedule-1,4c40f1f1c0a2e115,83dc5b6747b40cc6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:53.093 INFO 4486 --- [ocp-async-6,1deeb498b58f5b27,66f222a374a3efc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:53.465 INFO 4486 --- [ocp-schedules-28,a77d3003c4ae909c,c32ee02961bbecee] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:37:53.467 INFO 4486 --- [ocp-schedules-28,a77d3003c4ae909c,c32ee02961bbecee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:37:53.468 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:37:53.468 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:37:53.468 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:37:54.290 INFO 4486 --- [ocp-schedules-19,4c0213787700d073,b76d662a0877694e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:37:54.337 WARN 4486 --- [scanner-13259961-1,4c0213787700d073,3c0535599ec8f0c5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:37:54.819 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:55.093 INFO 4486 --- [ocp-async-10,3cb123f2914e21fa,1772bbf4ac875a43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:55.994 INFO 4486 --- [prometheus-minute-schedule-1,22ac26d2357464d8,5214e1613873754d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:56.001 INFO 4486 --- [prometheus-minute-schedule-1,22ac26d2357464d8,5214e1613873754d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:57.094 INFO 4486 --- [ocp-async-9,b4810906bf13d337,1fbe1430e87e501a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:57.280 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,374056c80e567444,9bc75e14b41ed25d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=374056c80e567444, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:57.280 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,374056c80e567444,9bc75e14b41ed25d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=374056c80e567444, duration=3 ms] 2024-10-16 13:37:57.821 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:37:58.353 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2a0fc8b14e665311,dcec037c50b806e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=2a0fc8b14e665311, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:37:58.354 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2a0fc8b14e665311,dcec037c50b806e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=2a0fc8b14e665311, duration=3 ms] 2024-10-16 13:37:58.919 INFO 4486 --- [ocp-schedules-27,194f862b05a2d21d,e6f476cefc785a7b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:37:58.994 INFO 4486 --- [prometheus-minute-schedule-1,c3f200d9805a53f9,76c95a029f9e2712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057080, interval=60 2024-10-16 13:37:59.001 INFO 4486 --- [prometheus-minute-schedule-1,c3f200d9805a53f9,76c95a029f9e2712] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:37:59.094 INFO 4486 --- [ocp-async-4,3a753d50d9c041d7,5ad8a4406e292771] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:37:59.154 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:37:59.173 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:37:59.175 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:37:59.197 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:37:59.198 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:37:59.222 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:37:59.225 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:37:59.245 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:37:59.247 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:37:59.271 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:37:59.272 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:37:59.297 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:37:59.299 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:37:59.323 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:37:59.325 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:37:59.348 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:37:59.349 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:37:59.374 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:37:59.375 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:37:59.397 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:37:59.398 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:37:59.421 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:37:59.423 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:37:59.449 INFO 4486 --- [ocp-schedules-18,56225854ff28d1db,7575b2626cc3d935] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:37:59.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,73962b8c4803d69c,39cd9b5fb66d6cd3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=73962b8c4803d69c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:37:59.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,73962b8c4803d69c,39cd9b5fb66d6cd3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=73962b8c4803d69c, duration=2 ms] 2024-10-16 13:38:00.455 INFO 4486 --- [ocp-schedules-17,097b1cabbd73cc23,f5ab2c544c51918c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:38:00.821 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:01.052 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,57e49545b87d12c1,4ddd340025ed1542] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=57e49545b87d12c1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:01.052 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,57e49545b87d12c1,4ddd340025ed1542] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=57e49545b87d12c1, duration=1 ms] 2024-10-16 13:38:01.094 INFO 4486 --- [ocp-async-6,ea0110ea34248c99,19b27e52b9da40a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:01.342 INFO 4486 --- [ocp-schedules-12,79d01d7186ef07ed,73ec22c8a063dc11] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:38:01.342+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:38:01.348 INFO 4486 --- [ocp-schedules-12,79d01d7186ef07ed,73ec22c8a063dc11] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:38:01.362 INFO 4486 --- [ocp-schedules-12,79d01d7186ef07ed,73ec22c8a063dc11] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:38:01.375 INFO 4486 --- [ocp-schedules-12,79d01d7186ef07ed,73ec22c8a063dc11] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:38:01.378 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,aa2a83e2e101f76b,bab5b3cc531f6a5e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=aa2a83e2e101f76b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:01.378 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,aa2a83e2e101f76b,bab5b3cc531f6a5e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=aa2a83e2e101f76b, duration=1 ms] 2024-10-16 13:38:01.994 INFO 4486 --- [prometheus-minute-schedule-1,2429c4e160e7c675,a78b402d01e23c4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:02.000 INFO 4486 --- [prometheus-minute-schedule-1,2429c4e160e7c675,a78b402d01e23c4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:02.396 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:38:02.396 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:38:02.402 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:38:02.403 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:38:02.403 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:38:02.403 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:38:02.407 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:38:02.408 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:38:02.408 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:38:02.408 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:38:02.453 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:38:02.461 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:38:02.461 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:38:02.461 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:38:02.461 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:38:02.461 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:38:02.467 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:38:02.863 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:38:02.994 INFO 4486 --- [ocp-async-8,7561e21523f214e0,448b04a1ad06302c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:38:03.094 INFO 4486 --- [ocp-async-9,0b7b0760a5d647fb,9d511aa9f42a4235] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:03.359 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.951375707S 2024-10-16 13:38:03.359 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:38:03.444 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:38:03.444 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:38:03.469 INFO 4486 --- [ocp-schedules-28,a77d3003c4ae909c,c32ee02961bbecee] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:38:03.472 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:38:03.562 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:38:03.562 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:38:03.562 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:38:03.562 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:38:03.562 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:38:03.563 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:38:03.571 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:38:03.700 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] o.s.boot.SpringApplication : Started application in 1.297 seconds (JVM running for 77986.751) 2024-10-16 13:38:03.700 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.298023243S 2024-10-16 13:38:03.822 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:04.039 INFO 4486 --- [ocp-schedules-18,c8e9a449757acac8,bccd65cd572be0cd] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:38:04.550 INFO 4486 --- [ocp-schedules-30,7225cc320fce4327,f6de3c4a339ab8da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:38:04.568 WARN 4486 --- [scanner-13260283-1,7225cc320fce4327,747146a9d0805042] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:38:04.620 INFO 4486 --- [alarm-detect-3,7225cc320fce4327,fd56a4c3c23cf6ab] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057050 ~ 1729057074 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057065, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057065, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057065=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:38:04.622 INFO 4486 --- [alarm-detect-3,7225cc320fce4327,fd56a4c3c23cf6ab] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:38:04.994 INFO 4486 --- [prometheus-minute-schedule-1,e1593b68cb6020b3,7ba638fa9deb769d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:05.000 INFO 4486 --- [prometheus-minute-schedule-1,e1593b68cb6020b3,7ba638fa9deb769d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:05.095 INFO 4486 --- [ocp-async-4,a6a71225afdf33b7,89707f9c25e5670f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:06.728 INFO 4486 --- [ocp-schedules-16,c5ed3c009139f197,b15b748251c273af] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:38:06.728 INFO 4486 --- [ocp-schedules-16,2d9465a3ef874030,af67271cb8ea2593] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:38:06.730 INFO 4486 --- [ocp-schedules-16,2d9465a3ef874030,af67271cb8ea2593] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:38:06.770 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.771 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:06.777 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.778 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:06.804 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.805 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:06.810 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:06.821 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.822 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:38:06.823 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:06.856 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:06.857 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.858 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:06.859 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:38:06.869 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.873 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:38:06.875 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:06.895 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:06.895 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:06.917 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:06.919 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7f74a1cca7780241,cc93051e58a5394c] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&version=2&ObClusterId=1701313637, client=10.197.49.184, traceId=7f74a1cca7780241, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=OceanBase,1701313637,2,{"ObRegion":"OceanBase","ObCluster":"OceanBase","ObRegionId":1701313637,"ObClusterId":1701313637,"Type":"PRIMARY","timestamp":1729057086911421,"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:38:06.924 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:06.925 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7f74a1cca7780241,cc93051e58a5394c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=OceanBase, obClusterId=1701313637, rootServerJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, groupName=OceanBase, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:38:06.925 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7f74a1cca7780241,cc93051e58a5394c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:38:06.925 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7f74a1cca7780241,cc93051e58a5394c] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=OceanBase, clusterId=1701313637, affectRows=1 2024-10-16 13:38:06.926 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7f74a1cca7780241,cc93051e58a5394c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.184, traceId=7f74a1cca7780241, duration=13 ms] 2024-10-16 13:38:06.926 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.927 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:38:06.928 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:06.929 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.929 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:06.930 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:38:06.938 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:38:06.943 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:06.946 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:06.968 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:38:06.968 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:38:06.973 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:38:06.985 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:06.987 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.988 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:06.991 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:06.995 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:38:06.999 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:07.059 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:38:07.066 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:38:07.067 INFO 4486 --- [ocp-schedules-7,07d73679f7d0242c,54877a7038d963e6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:38:07.069 INFO 4486 --- [ocp-schedules-7,07d73679f7d0242c,54877a7038d963e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:38:07.070 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:07.075 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:07.075 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:07.075 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:38:07.081 INFO 4486 --- [ocp-schedules-30,99b3a9f737ec7b60,2184eae19d9f8ede] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:38:07.083 INFO 4486 --- [ocp-schedules-30,99b3a9f737ec7b60,2184eae19d9f8ede] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:38:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:38:07.086 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:38:07.086 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056486917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:38:07.089 INFO 4486 --- [ocp-schedules-15,69b76f874b8a4517,9c42c6ccc0c15253] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:38:07.089 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:38:07.089 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:38:07.090 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:38:07.090 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:38:07.091 INFO 4486 --- [ocp-schedules-24,d2c848056497c59c,bbad76b578232fc2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:38:07.091 INFO 4486 --- [ocp-schedules-10,771014afbd8277fc,dc07883f487eed1e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:38:07.092 INFO 4486 --- [host-check-clockdiff1,d2c848056497c59c,292877c507764f8b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:38:07.092 INFO 4486 --- [host-check-clockdiff2,d2c848056497c59c,09840ccc93572db5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:38:07.093 INFO 4486 --- [host-check-agent-status1,771014afbd8277fc,44aedf4ec0a534cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:38:07.094 INFO 4486 --- [host-check-clockdiff4,d2c848056497c59c,307fdd3ddb17faa7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:38:07.095 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:07.095 INFO 4486 --- [host-check-clockdiff3,d2c848056497c59c,3ffa0acfd20cc3fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:38:07.096 INFO 4486 --- [host-check-agent-status2,771014afbd8277fc,ca738994d78b7968] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:38:07.096 INFO 4486 --- [ocp-async-2,f95fb5b7462b0b8f,5920b9e8abc449d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:07.096 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:38:07.097 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:38:07.098 INFO 4486 --- [host-check-agent-status3,771014afbd8277fc,44ffa2b0f2e3d379] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:38:07.098 INFO 4486 --- [host-check-agent-status4,771014afbd8277fc,26683882e016478b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:38:07.098 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:38:07.098 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:07.098 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:38:07.099 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:07.113 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:07.113 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.114 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:07.131 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:38:07.131 INFO 4486 --- [host-check-agent-status1,771014afbd8277fc,44aedf4ec0a534cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:07.131 INFO 4486 --- [host-check-agent-status2,771014afbd8277fc,ca738994d78b7968] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:38:07.145 INFO 4486 --- [host-check-clockdiff1,d2c848056497c59c,292877c507764f8b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057087 0 0 , error= 2024-10-16 13:38:07.152 INFO 4486 --- [host-check-clockdiff3,d2c848056497c59c,3ffa0acfd20cc3fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057087 3 3 , error= 2024-10-16 13:38:07.153 INFO 4486 --- [host-check-agent-status2,771014afbd8277fc,ca738994d78b7968] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:38:07.154 INFO 4486 --- [host-check-agent-status1,771014afbd8277fc,44aedf4ec0a534cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:07.158 INFO 4486 --- [host-check-agent-status3,771014afbd8277fc,44ffa2b0f2e3d379] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:38:07.159 INFO 4486 --- [host-check-agent-status4,771014afbd8277fc,26683882e016478b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:07.159 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:07.159 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:07.160 INFO 4486 --- [host-check-agent-status3,771014afbd8277fc,44ffa2b0f2e3d379] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:38:07.160 INFO 4486 --- [host-check-agent-status4,771014afbd8277fc,26683882e016478b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:07.173 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:38:07.174 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:38:07.174036944+08:00 2024-10-16 13:38:07.175 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:38:07.174+08:00 2024-10-16 13:38:07.179 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:38:07.182 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:38:07.185132009+08:00 2024-10-16 13:38:07.182 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:38:07.174+08:00 and status AVAILABLE 2024-10-16 13:38:07.182 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:38:07.183 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.183 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:38:07.182+08:00 2024-10-16 13:38:07.183 INFO 4486 --- [host-check-agent-status1,771014afbd8277fc,44aedf4ec0a534cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:38:07.183 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:38:07.184 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:07.185 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:38:07.185 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:38:07.187967499+08:00 2024-10-16 13:38:07.186 INFO 4486 --- [host-check-agent-status2,771014afbd8277fc,ca738994d78b7968] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:38:07.186 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:38:07.189069985+08:00 2024-10-16 13:38:07.186 INFO 4486 --- [host-check-agent-status4,771014afbd8277fc,26683882e016478b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:38:07.186 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:38:07.182+08:00 and status AVAILABLE 2024-10-16 13:38:07.187 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:38:07.186+08:00 2024-10-16 13:38:07.187 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:38:07.186+08:00 2024-10-16 13:38:07.188 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:38:07.189 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:38:07.186+08:00 and status AVAILABLE 2024-10-16 13:38:07.190 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:38:07.190 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:38:07.186+08:00 and status AVAILABLE 2024-10-16 13:38:07.191 INFO 4486 --- [host-check-agent-status4,771014afbd8277fc,26683882e016478b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:38:07.191 INFO 4486 --- [host-check-agent-status4,771014afbd8277fc,26683882e016478b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:38:07.191 INFO 4486 --- [host-check-agent-status4,771014afbd8277fc,26683882e016478b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:38:07.191 INFO 4486 --- [host-check-agent-status2,771014afbd8277fc,ca738994d78b7968] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:38:07.191 INFO 4486 --- [host-check-agent-status2,771014afbd8277fc,ca738994d78b7968] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:38:07.191 INFO 4486 --- [host-check-agent-status2,771014afbd8277fc,ca738994d78b7968] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:38:07.192 INFO 4486 --- [host-check-agent-status3,771014afbd8277fc,44ffa2b0f2e3d379] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:38:07.192 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:38:07.193 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:38:07.194 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:07.194 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:07.194 INFO 4486 --- [host-check-agent-status1,771014afbd8277fc,44aedf4ec0a534cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:38:07.194 INFO 4486 --- [host-check-agent-status1,771014afbd8277fc,44aedf4ec0a534cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:38:07.194 INFO 4486 --- [host-check-agent-status1,771014afbd8277fc,44aedf4ec0a534cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:38:07.195 INFO 4486 --- [host-check-agent-status3,771014afbd8277fc,44ffa2b0f2e3d379] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:38:07.196 INFO 4486 --- [host-check-agent-status3,771014afbd8277fc,44ffa2b0f2e3d379] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:38:07.196 INFO 4486 --- [host-check-agent-status3,771014afbd8277fc,44ffa2b0f2e3d379] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:38:07.196 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:07.197 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:07.198 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:38:07.200 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:38:07.201 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:38:07.203 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:38:07.204 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:07.206 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:07.207 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.212 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:38:07.221 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:07.223 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:07.223 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:07.223 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.223 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:07.224 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:07.226 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:07.232 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:07.234 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.235 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:07.249 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:07.249 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:07.249 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.251 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:07.253 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.254 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:38:07.262 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:38:07.263 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:38:07.265 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:38:07.270 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.271 INFO 4486 --- [ocp-schedules-25,27de9925d868747a,0af984442fd9fbd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:38:07.273 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:38:07.274 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:38:07.275 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:38:07.279 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:38:07.281 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:07.289 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:38:07.292 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cee34caf595efb4e,1539e489af9ae0f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cee34caf595efb4e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:07.292 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cee34caf595efb4e,1539e489af9ae0f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cee34caf595efb4e, duration=1 ms] 2024-10-16 13:38:07.301 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:07.305 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:07.306 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:07.308 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:07.308 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:07.310 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.311 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:07.315 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:38:07.319 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:38:07.324 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:38:07.326 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:38:07.334 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:38:07.344 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:38:07.345 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:38:07.348 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:38:07.352 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:38:07.355 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:38:07.359 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:38:07.364 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:07.366 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:38:07.374 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:07.377 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:38:07.377 WARN 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : Failed to get ob process info, errMsg:[AgentClient]:http request is failed, response:Cannot get process info of open /proc/1056867/status: no such file or directory, reason: %!v(MISSING) 2024-10-16 13:38:07.378 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:07.382 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 2024-10-16 13:38:07.388 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:07.389 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:07.391 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:38:07.394 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:07.401 INFO 4486 --- [host-check-host-status3,69b76f874b8a4517,0a493c429f307d20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:38:07.409 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:38:07.411 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:38:07.419 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:38:07.420 INFO 4486 --- [host-check-host-status4,69b76f874b8a4517,b60c0b36d338c16d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:38:07.429 INFO 4486 --- [host-check-host-status2,69b76f874b8a4517,1b43c4c46b706403] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:38:07.448 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:38:07.459 INFO 4486 --- [host-check-host-status1,69b76f874b8a4517,99e788d93322125f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:38:07.543 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 got ObProcessInfoResult(pid=91568, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:38:07.570 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:07.572 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:07.573 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:38:07.602 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:38:07.602 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:38:07.994 INFO 4486 --- [prometheus-minute-schedule-1,7edca8ce0e68fcf6,d33da42a45fb9bed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:08.000 INFO 4486 --- [prometheus-minute-schedule-1,7edca8ce0e68fcf6,d33da42a45fb9bed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:08.064 INFO 4486 --- [host-check-clockdiff2,d2c848056497c59c,09840ccc93572db5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057088 3 3 , error= 2024-10-16 13:38:08.162 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,180a15e60556d2a5,a5d5a7d3b1f15673] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=180a15e60556d2a5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:38:08.162 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,180a15e60556d2a5,a5d5a7d3b1f15673] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=180a15e60556d2a5, duration=1 ms] 2024-10-16 13:38:09.095 INFO 4486 --- [ocp-async-10,5ed5c7a6c8fa9901,88695b98bb129a5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:09.490 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2206359f6568751a,a258136a7c70603b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2206359f6568751a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:09.490 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2206359f6568751a,a258136a7c70603b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2206359f6568751a, duration=1 ms] 2024-10-16 13:38:09.825 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:10.007 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:38:10.026 INFO 4486 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:38:10.028 INFO 4486 --- [Check obproxy2,0dd0a5de091cb98c,56701808ffcf65c1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:38:10.029 INFO 4486 --- [Check obproxy1,e46aae24f974896d,cbe4745a2458ccc7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:38:10.035 INFO 4486 --- [Check obproxy2,0dd0a5de091cb98c,56701808ffcf65c1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:10.036 INFO 4486 --- [Check obproxy1,e46aae24f974896d,cbe4745a2458ccc7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:10.051 INFO 4486 --- [Collect obproxy1,8ce4f72699de69c4,14422ff7617af7be] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:38:10.051 INFO 4486 --- [Collect obproxy1,8ce4f72699de69c4,14422ff7617af7be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:10.053 INFO 4486 --- [Collect obproxy2,7534c6b89773cace,ec407b78a10ce800] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:38:10.053 INFO 4486 --- [Collect obproxy2,7534c6b89773cace,ec407b78a10ce800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:10.059 INFO 4486 --- [Check obproxy1,e46aae24f974896d,cbe4745a2458ccc7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:38:10.059 INFO 4486 --- [Check obproxy1,e46aae24f974896d,cbe4745a2458ccc7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:38:10.062 INFO 4486 --- [Check obproxy2,0dd0a5de091cb98c,56701808ffcf65c1] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:38:10.062 INFO 4486 --- [Check obproxy2,0dd0a5de091cb98c,56701808ffcf65c1] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:38:10.072 INFO 4486 --- [Collect obproxy2,7534c6b89773cace,ec407b78a10ce800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:38:10.077 INFO 4486 --- [Collect obproxy1,8ce4f72699de69c4,14422ff7617af7be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:38:10.077 INFO 4486 --- [Collect obproxy2,7534c6b89773cace,ec407b78a10ce800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:38:10.082 INFO 4486 --- [Collect obproxy2,7534c6b89773cace,ec407b78a10ce800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:38:10.083 INFO 4486 --- [Collect obproxy1,8ce4f72699de69c4,14422ff7617af7be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:38:10.088 INFO 4486 --- [Collect obproxy1,8ce4f72699de69c4,14422ff7617af7be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:38:10.460 INFO 4486 --- [ocp-schedules-1,9c376c353b73a375,72e16f1c8800053d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:38:10.979 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:38:10.994 INFO 4486 --- [prometheus-minute-schedule-1,e7f0e6cd8da55302,da0768ba6412d0b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:11.001 INFO 4486 --- [prometheus-minute-schedule-1,e7f0e6cd8da55302,da0768ba6412d0b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:38:11.057 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b65823f5a5fe7da2,26b1b158d5504316] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b65823f5a5fe7da2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:11.058 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b65823f5a5fe7da2,26b1b158d5504316] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b65823f5a5fe7da2, duration=2 ms] 2024-10-16 13:38:11.095 INFO 4486 --- [ocp-async-7,d52bdb308d57572a,54c49fd4bef28bd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:11.245 INFO 4486 --- [host-check-clockdiff4,d2c848056497c59c,307fdd3ddb17faa7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057091 3 3 , error= 2024-10-16 13:38:11.382 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,64a7f3bb2eec3c55,1113ea56505baf07] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=64a7f3bb2eec3c55, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:11.382 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,64a7f3bb2eec3c55,1113ea56505baf07] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=64a7f3bb2eec3c55, duration=1 ms] 2024-10-16 13:38:12.826 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:12.994 INFO 4486 --- [ocp-async-4,1e7033bbbd43de8a,bbabb866ec4bb6d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:38:13.096 INFO 4486 --- [ocp-async-6,78ad6ac6e9e688ac,bb47a4856b8ff5a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:13.430 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057020, nodeCount=733, affectRows=733 2024-10-16 13:38:13.433 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057020, nodeCount=732, validCount=732 2024-10-16 13:38:13.434 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057020, nodeCount=1000, affectRows=1000 2024-10-16 13:38:13.435 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057020, nodeCount=1000, affectRows=1000 2024-10-16 13:38:13.436 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057020, nodeCount=719, affectRows=719 2024-10-16 13:38:13.438 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057020, nodeCount=998, validCount=997 2024-10-16 13:38:13.439 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057020, nodeCount=719, validCount=719 2024-10-16 13:38:13.439 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057020, nodeCount=1000, validCount=1000 2024-10-16 13:38:13.994 INFO 4486 --- [prometheus-minute-schedule-1,3a2cfa81b2f93375,8285813ffd38672e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:14.000 INFO 4486 --- [prometheus-minute-schedule-1,3a2cfa81b2f93375,8285813ffd38672e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:14.585 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057020, nodeCount=477, affectRows=477 2024-10-16 13:38:14.698 INFO 4486 --- [ocp-schedules-20,b37c3d77a64c4fd3,da0643f75b41a8ca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:38:14.743 WARN 4486 --- [scanner-13261366-1,b37c3d77a64c4fd3,d185d24da7d098f6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:38:14.806 INFO 4486 --- [alarm-detect-2,b37c3d77a64c4fd3,217b8b3914d3999b] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057065 ~ 1729057084 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057080, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057080, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057080=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:38:15.096 INFO 4486 --- [ocp-async-5,274704a27e868a06,9dd74ccbabb3c61d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:15.310 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057020, nodeCount=477, validCount=477 2024-10-16 13:38:15.827 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:16.994 INFO 4486 --- [prometheus-minute-schedule-1,23990ee27768a53f,f2b0b20d44bf4883] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:17.000 INFO 4486 --- [prometheus-minute-schedule-1,23990ee27768a53f,f2b0b20d44bf4883] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:17.070 INFO 4486 --- [ocp-schedules-7,07d73679f7d0242c,54877a7038d963e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:38:17.096 INFO 4486 --- [ocp-async-3,a71d5cfe9e26bd72,9739a230c799de9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:17.117 INFO 4486 --- [ocp-schedules-6,4a8e8f6d25a93111,0662839b831f03e0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:38:17.189 INFO 4486 --- [ocp-schedules-30,99b3a9f737ec7b60,2184eae19d9f8ede] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:38:17.296 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9b61217acc54653a,7b57bb6ffa975a58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9b61217acc54653a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:17.296 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9b61217acc54653a,7b57bb6ffa975a58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9b61217acc54653a, duration=1 ms] 2024-10-16 13:38:17.550 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c237c2f0d23ff1ca,e15cdd47f014acc6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=c237c2f0d23ff1ca, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:38:17.551 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c237c2f0d23ff1ca,e15cdd47f014acc6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=c237c2f0d23ff1ca, duration=2 ms] 2024-10-16 13:38:18.829 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:18.923 INFO 4486 --- [ocp-schedules-23,51ea6e03111678c1,ae6dc165efc0989e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:38:19.096 INFO 4486 --- [ocp-async-1,fb821000f16f542b,0f6daad7e79b8281] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:19.496 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c52e4f712814ec4e,6256e59ca7e9cd17] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c52e4f712814ec4e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:19.497 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c52e4f712814ec4e,6256e59ca7e9cd17] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c52e4f712814ec4e, duration=1 ms] 2024-10-16 13:38:19.994 INFO 4486 --- [prometheus-minute-schedule-1,9d28294acd03d13e,fe8906fde0485ab0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:19.999 INFO 4486 --- [prometheus-minute-schedule-1,9d28294acd03d13e,fe8906fde0485ab0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:20.466 INFO 4486 --- [ocp-schedules-26,c34fae0c38efd112,93e7dec194297a13] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:38:20.811 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:38:20.996 INFO 4486 --- [prometheus-cache-monitor1,acc2eacd209ee6d0,0768eec9a07f7315] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13858,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:38:20.996 INFO 4486 --- [prometheus-cache-monitor1,acc2eacd209ee6d0,0768eec9a07f7315] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:38:21.062 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bc9894b9df634850,4ddff14e8011d39b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bc9894b9df634850, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:21.063 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bc9894b9df634850,4ddff14e8011d39b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bc9894b9df634850, duration=1 ms] 2024-10-16 13:38:21.096 INFO 4486 --- [ocp-async-2,d43df3c844e5318a,2d57930349fb51f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:21.386 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,434b21359f8a4d26,e688961da2438d0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=434b21359f8a4d26, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:21.387 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,434b21359f8a4d26,e688961da2438d0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=434b21359f8a4d26, duration=2 ms] 2024-10-16 13:38:21.830 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:22.902 INFO 4486 --- [ocp-schedules-6,1388363cb238fcf0,397545f33f2b4993] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:38:22.994 INFO 4486 --- [prometheus-minute-schedule-1,9180d9785e9315b1,849543fe4ca92eb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:22.995 INFO 4486 --- [ocp-async-2,f8e435f76481356e,82b18d65d23331ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:38:23.002 INFO 4486 --- [prometheus-minute-schedule-1,9180d9785e9315b1,849543fe4ca92eb3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:23.097 INFO 4486 --- [ocp-async-7,c4fd17d654ec4e69,759c06f539267286] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:24.389 INFO 4486 --- [ocp-schedules-13,59e3e9b63b32ec0e,0de101f618f301d5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:38:24.724 INFO 4486 --- [ocp-schedules-14,284720db29433d78,1db4547dbd941256] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:38:24.725 INFO 4486 --- [ocp-schedules-14,284720db29433d78,1db4547dbd941256] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:38:24.764 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:38:24.765 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.769 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:38:24.777 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:38:24.777 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.777 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.777 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:38:24.782 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:38:24.786 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.787 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:38:24.794 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.795 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:38:24.802 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.803 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:38:24.804 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.805 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:38:24.812 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.812 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:24.819 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.819 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:38:24.820 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.820 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.821 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:24.824 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:38:24.826 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:38:24.828 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.829 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:38:24.829 INFO 4486 --- [ocp-schedules-10,46077d7c58d675af,1b4328c8b9486e8d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:38:24.829 INFO 4486 --- [ocp-schedules-10,46077d7c58d675af,1b4328c8b9486e8d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:38:24.831 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:24.836 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.837 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:38:24.838 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.839 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:38:24.844 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.845 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:24.853 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.854 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:38:24.858 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.858 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.859 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:24.859 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.860 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.861 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.862 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.863 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.864 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.865 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.865 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.865 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.866 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:38:24.866 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.867 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.868 INFO 4486 --- [Collect all tenants major compactions1,46077d7c58d675af,74354a6ac5e6a71c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:38:24.868 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.868 INFO 4486 --- [Collect all tenants major compactions2,46077d7c58d675af,63d679283e51234c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:38:24.868 INFO 4486 --- [Collect all tenants major compactions3,46077d7c58d675af,303e855eec8871d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:38:24.869 INFO 4486 --- [Collect all tenants major compactions5,46077d7c58d675af,8b95ee67b88d9abc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:38:24.869 INFO 4486 --- [Collect all tenants major compactions4,46077d7c58d675af,7cb2f894a4e74776] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:38:24.869 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.869 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.869 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.870 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:38:24.870 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.870 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:38:24.871 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.872 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.872 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.873 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.874 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.874 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.875 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.876 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.877 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.878 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.878 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:38:24.881 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.881 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.881 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:38:24.882 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:38:24.888 INFO 4486 --- [ocp-schedules-25,c932f53f0881eca3,60be39a129939c38] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:38:24.889 INFO 4486 --- [Collect all tenants major compactions1,46077d7c58d675af,74354a6ac5e6a71c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.890 INFO 4486 --- [Collect all tenants major compactions2,46077d7c58d675af,63d679283e51234c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.890 INFO 4486 --- [Collect all tenants major compactions1,46077d7c58d675af,74354a6ac5e6a71c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:38:24.902 INFO 4486 --- [Collect all tenants major compactions2,46077d7c58d675af,63d679283e51234c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:38:24.903 INFO 4486 --- [Collect all tenants major compactions3,46077d7c58d675af,303e855eec8871d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.905 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:38:24.905 INFO 4486 --- [Collect all tenants major compactions3,46077d7c58d675af,303e855eec8871d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:38:24.906 INFO 4486 --- [Collect all tenants major compactions2,46077d7c58d675af,63d679283e51234c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:38:24.906 INFO 4486 --- [Collect all tenants major compactions1,46077d7c58d675af,74354a6ac5e6a71c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:38:24.907 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.908 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:38:24.909 INFO 4486 --- [Collect all tenants major compactions3,46077d7c58d675af,303e855eec8871d4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:38:24.912 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.913 INFO 4486 --- [Collect all tenants major compactions5,46077d7c58d675af,8b95ee67b88d9abc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.914 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:24.915 INFO 4486 --- [Collect all tenants major compactions5,46077d7c58d675af,8b95ee67b88d9abc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:38:24.916 INFO 4486 --- [Collect all tenants major compactions4,46077d7c58d675af,7cb2f894a4e74776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.916 INFO 4486 --- [Collect all tenants major compactions1,46077d7c58d675af,74354a6ac5e6a71c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:38:24.918 INFO 4486 --- [Collect all tenants major compactions4,46077d7c58d675af,7cb2f894a4e74776] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:38:24.918 INFO 4486 --- [Collect all tenants major compactions2,46077d7c58d675af,63d679283e51234c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:38:24.920 INFO 4486 --- [Collect all tenants major compactions3,46077d7c58d675af,303e855eec8871d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:38:24.921 INFO 4486 --- [Collect all tenants major compactions5,46077d7c58d675af,8b95ee67b88d9abc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:38:24.921 INFO 4486 --- [Collect all tenants major compactions5,46077d7c58d675af,8b95ee67b88d9abc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:38:24.922 INFO 4486 --- [Collect all tenants major compactions4,46077d7c58d675af,7cb2f894a4e74776] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:38:24.922 INFO 4486 --- [Collect all tenants major compactions4,46077d7c58d675af,7cb2f894a4e74776] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:38:24.924 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.926 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.926 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.927 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:24.930 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.930 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:24.931 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.932 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.932 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:38:24.934 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.935 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.938 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.939 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:24.939 WARN 4486 --- [scanner-13261682-1,c932f53f0881eca3,7e166aa9381c4660] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:38:24.941 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.942 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.945 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:24.946 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:38:24.948 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.950 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.952 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:38:24.952 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:38:24.955 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.956 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.958 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.962 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.964 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.965 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.968 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:38:24.972 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:38:24.977 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.978 INFO 4486 --- [ocp-schedules-10,46077d7c58d675af,1b4328c8b9486e8d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:38:24.978 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.981 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.983 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:24.986 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.988 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:24.991 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.993 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:24.996 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:24.997 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:25.000 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:25.002 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:25.005 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:25.006 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:38:25.009 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:25.010 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:38:25.011 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:25.013 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:25.013 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:25.014 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:38:25.014 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:38:25.016 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:38:25.019 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:38:25.024 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:25.024 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:38:25.024 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:25.024 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:38:25.024 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:25.024 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:38:25.028 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:38:25.028 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:38:25.082 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:25.084 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:25.085 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:38:25.097 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:25.097 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:38:25.097 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:25.097 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:38:25.097 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:38:25.097 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:38:25.097 INFO 4486 --- [ocp-async-4,0df0db551d0ec382,a252976e8dfcdd30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:25.101 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:38:25.101 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:38:25.994 INFO 4486 --- [prometheus-minute-schedule-1,f278a900de067299,e660f12f4e28ce27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:26.000 INFO 4486 --- [prometheus-minute-schedule-1,f278a900de067299,e660f12f4e28ce27] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:27.049 INFO 4486 --- [ocp-schedules-28,07ffe4b543bb9ead,c7014b103cc803f7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:38:27.049+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:38:27.050 INFO 4486 --- [ocp-schedules-28,07ffe4b543bb9ead,c7014b103cc803f7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:38:27.051 INFO 4486 --- [ocp-schedules-28,07ffe4b543bb9ead,c7014b103cc803f7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:38:27.052 INFO 4486 --- [ocp-schedules-28,07ffe4b543bb9ead,c7014b103cc803f7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:38:27.097 INFO 4486 --- [ocp-async-6,c2a6011ca7d1d6fc,90af6ca6b09b830d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:27.303 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d262595cdde42a11,036f8880debc5067] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d262595cdde42a11, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:27.303 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d262595cdde42a11,036f8880debc5067] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d262595cdde42a11, duration=2 ms] 2024-10-16 13:38:27.832 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:28.994 INFO 4486 --- [prometheus-minute-schedule-1,d4d78ec0a136a00f,c97e1db987bf90a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:29.000 INFO 4486 --- [prometheus-minute-schedule-1,d4d78ec0a136a00f,c97e1db987bf90a6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:29.097 INFO 4486 --- [ocp-async-5,5db297399a8f0fbb,f776ee47d88ca749] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:29.502 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,559f18805cd19dae,e2a6b89db8b13bf4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=559f18805cd19dae, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:29.503 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,559f18805cd19dae,e2a6b89db8b13bf4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=559f18805cd19dae, duration=2 ms] 2024-10-16 13:38:30.103 INFO 4486 --- [prometheus-second-schedule-1,28264b9f22683429,947b9d9ed2542173] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:38:30.471 INFO 4486 --- [ocp-schedules-10,0035e88118aa9571,1003ca24979813d7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:38:30.833 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:31.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ae2c0e991e886273,4f79e3aaafe7c810] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ae2c0e991e886273, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:31.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ae2c0e991e886273,4f79e3aaafe7c810] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ae2c0e991e886273, duration=1 ms] 2024-10-16 13:38:31.098 INFO 4486 --- [ocp-async-3,cba092ddb2d5e7d7,6a6b92bfbfee48ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:31.391 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4498870c379e5e9e,2b4244041006a7b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4498870c379e5e9e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:31.391 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4498870c379e5e9e,2b4244041006a7b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4498870c379e5e9e, duration=1 ms] 2024-10-16 13:38:31.994 INFO 4486 --- [prometheus-minute-schedule-1,9c0f1b5bc70aca21,dad5e68c5c8c0647] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:32.000 INFO 4486 --- [prometheus-minute-schedule-1,9c0f1b5bc70aca21,dad5e68c5c8c0647] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:32.995 INFO 4486 --- [ocp-async-4,643d016f0444291a,26397dc7ea058f1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:38:33.098 INFO 4486 --- [ocp-async-10,076e79e3a6d43162,523762657194af74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:33.834 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:34.994 INFO 4486 --- [prometheus-minute-schedule-1,52e7c4655676ddcf,a8fe11e320618f13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:35.000 INFO 4486 --- [prometheus-minute-schedule-1,52e7c4655676ddcf,a8fe11e320618f13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:35.058 INFO 4486 --- [ocp-schedules-12,bab4e5e31928f3af,384aef01434cc8c8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:38:35.098 WARN 4486 --- [scanner-13261928-1,bab4e5e31928f3af,a02f915fcffa9f2c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:38:35.099 INFO 4486 --- [ocp-async-9,8130ed08e58da434,ea7146f0cf515299] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:35.192 INFO 4486 --- [alarm-detect-18,bab4e5e31928f3af,9c3dd2e02b9d3b85] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057080 ~ 1729057105 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057095, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057095, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057095=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:38:36.835 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:37.065 INFO 4486 --- [ocp-schedules-13,c2d3275d9f5e5e92,165c446285d9ec88] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:38:37.066 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:38:37.066 INFO 4486 --- [ocp-schedules-5,cf58b0cd51de4f07,58e3654871cca487] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:38:37.068 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:38:37.068 INFO 4486 --- [ocp-schedules-5,cf58b0cd51de4f07,58e3654871cca487] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:38:37.071 INFO 4486 --- [ocp-schedules-13,c2d3275d9f5e5e92,165c446285d9ec88] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:38:37.072 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:38:37.082 INFO 4486 --- [ocp-schedules-9,51cc85004d9caaa4,282c334fd43b12ff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:38:37.084 INFO 4486 --- [ocp-schedules-9,51cc85004d9caaa4,282c334fd43b12ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:38:37.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:38:37.090 INFO 4486 --- [ocp-schedules-20,7b2dfd3decad1ccf,8dba825d38e65871] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:38:37.090 INFO 4486 --- [ocp-schedules-21,c58fca164fa49ba1,75b40fd4792afe72] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:38:37.090 INFO 4486 --- [ocp-schedules-29,9a6988eab36ac259,8a76528debe19386] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:38:37.090 INFO 4486 --- [host-check-agent-status1,c58fca164fa49ba1,e58dda62f880a6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:38:37.091 INFO 4486 --- [host-check-agent-status2,c58fca164fa49ba1,6a036b3a27cd2dce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:38:37.091 INFO 4486 --- [host-check-agent-status3,c58fca164fa49ba1,1ce1fc370ba7ee04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:38:37.092 INFO 4486 --- [host-check-clockdiff1,7b2dfd3decad1ccf,7280a2b0331c4981] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:38:37.092 INFO 4486 --- [host-check-clockdiff2,7b2dfd3decad1ccf,0e0189de311a370f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:38:37.092 INFO 4486 --- [host-check-clockdiff3,7b2dfd3decad1ccf,7665a74cc19e4b23] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:38:37.094 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:38:37.094 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:38:37.095 INFO 4486 --- [host-check-agent-status4,c58fca164fa49ba1,ba8b09fde8698c3e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:38:37.095 INFO 4486 --- [host-check-clockdiff4,7b2dfd3decad1ccf,0b008fcd76d312ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:38:37.096 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:38:37.096 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:38:37.099 INFO 4486 --- [ocp-async-4,87f2b516d7489029,32ecd20aa7e5d52b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:37.101 INFO 4486 --- [host-check-agent-status2,c58fca164fa49ba1,6a036b3a27cd2dce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:38:37.102 INFO 4486 --- [host-check-agent-status3,c58fca164fa49ba1,1ce1fc370ba7ee04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:38:37.102 INFO 4486 --- [host-check-agent-status4,c58fca164fa49ba1,ba8b09fde8698c3e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:37.102 INFO 4486 --- [host-check-agent-status1,c58fca164fa49ba1,e58dda62f880a6a0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:37.102 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:38:37.103 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:37.104 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:38:37.104 INFO 4486 --- [host-check-agent-status4,c58fca164fa49ba1,ba8b09fde8698c3e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:37.104 INFO 4486 --- [host-check-agent-status3,c58fca164fa49ba1,1ce1fc370ba7ee04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:38:37.104 INFO 4486 --- [host-check-agent-status2,c58fca164fa49ba1,6a036b3a27cd2dce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:38:37.104 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:37.105 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:37.105 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:38:37.105 INFO 4486 --- [host-check-agent-status1,c58fca164fa49ba1,e58dda62f880a6a0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:37.106 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:38:37.106 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:37.122 INFO 4486 --- [host-check-clockdiff4,7b2dfd3decad1ccf,0b008fcd76d312ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057117 3 3 , error= 2024-10-16 13:38:37.122 INFO 4486 --- [host-check-clockdiff1,7b2dfd3decad1ccf,7280a2b0331c4981] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057117 0 0 , error= 2024-10-16 13:38:37.122 INFO 4486 --- [host-check-clockdiff3,7b2dfd3decad1ccf,7665a74cc19e4b23] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057117 3 3 , error= 2024-10-16 13:38:37.133 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.133 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:37.135 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:38:37.150 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:38:37.151 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:38:37.151 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:38:37.153 INFO 4486 --- [host-check-agent-status2,c58fca164fa49ba1,6a036b3a27cd2dce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:38:37.154 INFO 4486 --- [host-check-agent-status4,c58fca164fa49ba1,ba8b09fde8698c3e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:38:37.155 INFO 4486 --- [host-check-agent-status3,c58fca164fa49ba1,1ce1fc370ba7ee04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:38:37.168 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.168 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:38:37.158844766+08:00 2024-10-16 13:38:37.168 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:38:37.159234264+08:00 2024-10-16 13:38:37.168 INFO 4486 --- [host-check-agent-status1,c58fca164fa49ba1,e58dda62f880a6a0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:38:37.168 INFO 4486 --- [host-check-agent-status2,c58fca164fa49ba1,6a036b3a27cd2dce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:38:37.168 INFO 4486 --- [host-check-agent-status2,c58fca164fa49ba1,6a036b3a27cd2dce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:38:37.168 INFO 4486 --- [host-check-agent-status2,c58fca164fa49ba1,6a036b3a27cd2dce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:38:37.168 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:38:37.159045044+08:00 2024-10-16 13:38:37.169 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:38:37.168+08:00 2024-10-16 13:38:37.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:37.169 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:38:37.168+08:00 2024-10-16 13:38:37.171 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:38:37.171 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:38:37.169+08:00 2024-10-16 13:38:37.172 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.172 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:38:37.168+08:00 and status AVAILABLE 2024-10-16 13:38:37.172 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:38:37.168+08:00 and status AVAILABLE 2024-10-16 13:38:37.172 INFO 4486 --- [host-check-agent-status3,c58fca164fa49ba1,1ce1fc370ba7ee04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:38:37.172 INFO 4486 --- [host-check-agent-status3,c58fca164fa49ba1,1ce1fc370ba7ee04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:38:37.172 INFO 4486 --- [host-check-agent-status3,c58fca164fa49ba1,1ce1fc370ba7ee04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:38:37.173 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:38:37.173484518+08:00 2024-10-16 13:38:37.174 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.174 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:38:37.174 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:38:37.174+08:00 2024-10-16 13:38:37.174 INFO 4486 --- [host-check-agent-status1,c58fca164fa49ba1,e58dda62f880a6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:38:37.174 INFO 4486 --- [host-check-agent-status1,c58fca164fa49ba1,e58dda62f880a6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:38:37.174 INFO 4486 --- [host-check-agent-status1,c58fca164fa49ba1,e58dda62f880a6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:38:37.175 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:38:37.175 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:38:37.175 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:38:37.169+08:00 and status AVAILABLE 2024-10-16 13:38:37.177 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:38:37.177 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.177 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:38:37.174+08:00 and status AVAILABLE 2024-10-16 13:38:37.179 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:38:37.181 INFO 4486 --- [host-check-agent-status4,c58fca164fa49ba1,ba8b09fde8698c3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:38:37.181 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:38:37.181 INFO 4486 --- [host-check-agent-status4,c58fca164fa49ba1,ba8b09fde8698c3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:38:37.181 INFO 4486 --- [host-check-agent-status4,c58fca164fa49ba1,ba8b09fde8698c3e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:38:37.183 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.184 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:38:37.184 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:38:37.185 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:37.186 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.186 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:38:37.186 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.186 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:37.187 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:38:37.187 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:38:37.189 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:37.189 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:38:37.189 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.190 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:37.190 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:37.198 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.199 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.199 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.199 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.199 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.199 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.200 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.201 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:37.208 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:37.214 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.215 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:37.215 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:38:37.218 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:37.221 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.223 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:38:37.223 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:38:37.223 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:38:37.237 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.238 INFO 4486 --- [ocp-schedules-26,79ea443c3bb5c530,20e0b8ba32ca4198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:38:37.250 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:38:37.250 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:38:37.258 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:38:37.262 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:38:37.262 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:38:37.268 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:38:37.278 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:37.280 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:37.284 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:37.285 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:38:37.297 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:38:37.305 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:38:37.309 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dbea98a2db2c8117,5d974b6e3aaa7277] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dbea98a2db2c8117, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:37.309 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dbea98a2db2c8117,5d974b6e3aaa7277] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dbea98a2db2c8117, duration=2 ms] 2024-10-16 13:38:37.313 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:38:37.315 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:38:37.317 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:38:37.322 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:38:37.327 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:38:37.332 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:37.332 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:38:37.346 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:37.352 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:37.360 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:38:37.368 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:38:37.371 INFO 4486 --- [host-check-host-status3,9a6988eab36ac259,b8353a7dd2e7b40b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:38:37.378 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:38:37.379 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:38:37.393 INFO 4486 --- [host-check-host-status4,9a6988eab36ac259,b238c27b66b18695] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:38:37.393 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:38:37.396 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:38:37.410 INFO 4486 --- [host-check-host-status2,9a6988eab36ac259,0d51100ac4546d52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:38:37.451 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:38:37.457 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.462 INFO 4486 --- [host-check-host-status1,9a6988eab36ac259,2933db558bd680cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:38:37.466 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.467 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:38:37.468 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.469 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.470 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:38:37.482 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.483 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.483 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.483 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.483 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.484 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.512 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:38:37.559 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:38:37.603 INFO 4486 --- [ocp-schedules-16,2d9465a3ef874030,af67271cb8ea2593] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:38:37.603 INFO 4486 --- [ocp-schedules-16,22c435451135eab3,0e00264ea2db45d7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:38:37.604 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.605 INFO 4486 --- [ocp-schedules-16,22c435451135eab3,0e00264ea2db45d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:38:37.621 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.623 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.624 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:38:37.625 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:38:37.629 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.630 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:37.632 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.633 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:38:37.634 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.635 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.636 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:38:37.645 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.647 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:37.648 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.648 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.648 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.648 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.648 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.648 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.655 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:38:37.664 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.665 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.665 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:38:37.670 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.671 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:37.705 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.706 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.706 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:37.707 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:38:37.708 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.711 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:38:37.712 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:37.722 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:37.722 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:37.732 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:37.743 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.744 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.744 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:37.745 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:38:37.750 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:38:37.754 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:37.756 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:37.776 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:38:37.788 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.838 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.839 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:38:37.852 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:38:37.860 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:38:37.861 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056540308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:38:37.861 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.863 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.863 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:38:37.872 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:38:37.872 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:38:37.889 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.891 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.892 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:38:37.895 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.898 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:38:37.902 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:37.909 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:38:37.931 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:37.931 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:37.954 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:37.956 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.957 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:38:37.958 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:37.958 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.958 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:37.958 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.959 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:38:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:38:37.967 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.967 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.967 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.967 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.967 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:37.968 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:37.984 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:37.984 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:37.987 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:38:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:38:37.994 INFO 4486 --- [prometheus-minute-schedule-1,58e8fae2465a1101,48ce02145d98f0f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:37.999 INFO 4486 --- [prometheus-minute-schedule-1,58e8fae2465a1101,48ce02145d98f0f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:38.013 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:38.013 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:38:38.042 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:38.042 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:38:38.065 INFO 4486 --- [host-check-clockdiff2,7b2dfd3decad1ccf,0e0189de311a370f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057118 3 3 , error= 2024-10-16 13:38:38.083 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:38:38.111 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:38.112 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:38.113 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:38:38.117 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:38:38.122 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:38:38.126 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:38:38.128 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:38:38.152 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:38:38.210 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:38:38.259 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:38.261 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:38.262 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:38:38.263 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:38.264 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:38.264 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:38:38.273 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:38.273 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:38.273 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:38.273 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:38.273 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:38:38.274 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:38.291 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:38:38.321 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:38:38.322 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:38:38.323 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:38:38.337 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:38:38.350 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:38:38.350 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:38:38.439 INFO 4486 --- [ocp-schedules-21,1d0d3931cc6ae196,1bad9f485c5d246c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:38:38.439 INFO 4486 --- [ocp-schedules-21,1d0d3931cc6ae196,1bad9f485c5d246c] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:38:38.440 INFO 4486 --- [ocp-schedules-21,1d0d3931cc6ae196,1bad9f485c5d246c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:38:38.528 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:38:38.340701, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:38:38.529 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:38:38.340701, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:38:38.571 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:38:38.340701} } 2024-10-16 13:38:38.632 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:38:38.640 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:38:38.644 INFO 4486 --- [ocp-schedules-4,9686f7cfe9068f75,dff8396c48adef97] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:38:38.928 INFO 4486 --- [ocp-schedules-6,34d699a7564d50a5,c2d181dfb194ec96] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:38:39.099 INFO 4486 --- [ocp-async-10,ba1d96c9f54f3eae,d33a599e891177b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:39.508 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,868bac705da8f904,d004d00a8f749e56] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=868bac705da8f904, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:39.508 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,868bac705da8f904,d004d00a8f749e56] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=868bac705da8f904, duration=2 ms] 2024-10-16 13:38:39.837 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:40.006 INFO 4486 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:38:40.007 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:38:40.026 INFO 4486 --- [Check obproxy1,7926bb5148d56e48,5e3547969e445a2a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:38:40.026 INFO 4486 --- [Check obproxy2,65f661a813c112f6,3aa40f2ae01809d9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:38:40.026 INFO 4486 --- [Check obproxy2,65f661a813c112f6,3aa40f2ae01809d9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:40.026 INFO 4486 --- [Check obproxy1,7926bb5148d56e48,5e3547969e445a2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:40.032 INFO 4486 --- [Collect obproxy2,2f52f120f4298621,1af6f77b2fe98398] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:38:40.033 INFO 4486 --- [Collect obproxy2,2f52f120f4298621,1af6f77b2fe98398] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:40.033 INFO 4486 --- [Collect obproxy1,ad28ce419121e2ec,b630e158be69d8ae] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:38:40.033 INFO 4486 --- [Collect obproxy1,ad28ce419121e2ec,b630e158be69d8ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:38:40.049 INFO 4486 --- [Check obproxy2,65f661a813c112f6,3aa40f2ae01809d9] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:38:40.049 INFO 4486 --- [Check obproxy2,65f661a813c112f6,3aa40f2ae01809d9] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:38:40.050 INFO 4486 --- [Check obproxy1,7926bb5148d56e48,5e3547969e445a2a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:38:40.050 INFO 4486 --- [Check obproxy1,7926bb5148d56e48,5e3547969e445a2a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:38:40.053 INFO 4486 --- [Collect obproxy2,2f52f120f4298621,1af6f77b2fe98398] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:38:40.059 INFO 4486 --- [Collect obproxy2,2f52f120f4298621,1af6f77b2fe98398] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:38:40.063 INFO 4486 --- [Collect obproxy1,ad28ce419121e2ec,b630e158be69d8ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:38:40.064 INFO 4486 --- [Collect obproxy2,2f52f120f4298621,1af6f77b2fe98398] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:38:40.066 INFO 4486 --- [Collect obproxy1,ad28ce419121e2ec,b630e158be69d8ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:38:40.069 INFO 4486 --- [Collect obproxy1,ad28ce419121e2ec,b630e158be69d8ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:38:40.478 INFO 4486 --- [ocp-schedules-18,c3f597a0533c29e7,3987771e2886594f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:38:40.994 INFO 4486 --- [prometheus-minute-schedule-1,0b8486461a71a9b5,1cd9d229a32be4ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:40.995 INFO 4486 --- [prometheus-hour-schedule-1,f9d80eb18c160a42,8a6e7dde3f48d600] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:38:41.000 INFO 4486 --- [prometheus-minute-schedule-1,0b8486461a71a9b5,1cd9d229a32be4ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:41.074 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,1c47514404797e47,fcca9ab62a45e128] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1c47514404797e47, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:41.074 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,1c47514404797e47,fcca9ab62a45e128] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1c47514404797e47, duration=1 ms] 2024-10-16 13:38:41.100 INFO 4486 --- [ocp-async-9,29b8406fdf6683f6,26880b6127b0e1a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:41.395 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1ddf7fdafd707cca,cf5997e5fed6bb10] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1ddf7fdafd707cca, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:41.395 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1ddf7fdafd707cca,cf5997e5fed6bb10] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1ddf7fdafd707cca, duration=1 ms] 2024-10-16 13:38:42.100 INFO 4486 --- [ocp-schedules-1,ca028f82f5a32228,ed4eb4f632cfb509] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:38:42.838 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:42.995 INFO 4486 --- [ocp-async-8,992ac7a6271bce8b,f7fa0af985ec7247] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:38:43.100 INFO 4486 --- [ocp-async-4,73c0f93efc49f2ce,294388b3e13324ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:43.994 INFO 4486 --- [prometheus-minute-schedule-1,428958ce644e4028,01ed4f70f35123b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:44.000 INFO 4486 --- [prometheus-minute-schedule-1,428958ce644e4028,01ed4f70f35123b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:45.100 INFO 4486 --- [ocp-async-10,62bc7f4bd0284fa0,21b5a86f42a04479] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:45.282 INFO 4486 --- [ocp-schedules-10,ae6eb2399e0ceb1d,cd282b4f2795c62d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:38:45.322 WARN 4486 --- [scanner-13262294-1,ae6eb2399e0ceb1d,659d78378e78ebaf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:38:45.358 INFO 4486 --- [alarm-detect-1,ae6eb2399e0ceb1d,efbef9336d16cc2d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057095 ~ 1729057115 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057110, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057110, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057110=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:38:45.471 INFO 4486 --- [ocp-schedules-29,73b65257ef8d558f,4a3ac43c6f311f13] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:38:45.476 INFO 4486 --- [ocp-schedules-29,73b65257ef8d558f,4a3ac43c6f311f13] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:38:45.476 INFO 4486 --- [ocp-schedules-29,73b65257ef8d558f,4a3ac43c6f311f13] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:38:45.490 INFO 4486 --- [ocp-schedules-7,e76a28d292c64dce,f38298f64faca0b3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:38:45.490 INFO 4486 --- [ocp-schedules-7,e76a28d292c64dce,f38298f64faca0b3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:38:45.839 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:46.994 INFO 4486 --- [prometheus-minute-schedule-1,45f413712be1704a,2ac860c06a62a073] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:47.001 INFO 4486 --- [prometheus-minute-schedule-1,45f413712be1704a,2ac860c06a62a073] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:47.070 INFO 4486 --- [ocp-schedules-5,cf58b0cd51de4f07,58e3654871cca487] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:38:47.100 INFO 4486 --- [ocp-async-7,e38de3005143c19a,9fb4b02cb2c0a871] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:47.119 INFO 4486 --- [ocp-schedules-4,d1586398de557fc7,175be9b72bca1849] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:38:47.175 INFO 4486 --- [ocp-schedules-9,51cc85004d9caaa4,282c334fd43b12ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:38:47.314 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b2b2d8f10185f928,6c8ccd1b4a9743d5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b2b2d8f10185f928, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:47.314 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b2b2d8f10185f928,6c8ccd1b4a9743d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b2b2d8f10185f928, duration=1 ms] 2024-10-16 13:38:48.839 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:49.100 INFO 4486 --- [ocp-async-1,54456d3b6cd0e070,fb1e66f17f33075e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:49.513 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,db943edd84af95d1,041c4f34e3d8ff34] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=db943edd84af95d1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:49.513 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,db943edd84af95d1,041c4f34e3d8ff34] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=db943edd84af95d1, duration=1 ms] 2024-10-16 13:38:49.994 INFO 4486 --- [prometheus-minute-schedule-1,dbb8b2a8d134b508,a1be985662eb428d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:50.001 INFO 4486 --- [prometheus-minute-schedule-1,dbb8b2a8d134b508,a1be985662eb428d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:50.241 INFO 4486 --- [ocp-schedules-29,0f4bdc56b052899e,41a0422602d02892] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:38:50.242 INFO 4486 --- [ocp-schedules-29,0f4bdc56b052899e,41a0422602d02892] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:38:50.242 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:38:50.242 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:38:50.242 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:38:50.242 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:38:50.242 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:38:50.247 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:38:50.247 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:38:50.247 INFO 4486 --- [ocp-schedules-29,0f4bdc56b052899e,41a0422602d02892] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:38:50.248 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:38:50.248 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:38:50.248 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:38:50.249 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:38:50.249 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:38:50.249 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:38:50.249 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:38:50.249 INFO 4486 --- [ocp-schedules-29,0f4bdc56b052899e,41a0422602d02892] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:38:50.251 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:38:50.274 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:38:50.275 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:38:50.275 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:38:50.277 INFO 4486 --- [ocp-schedules-29,0f4bdc56b052899e,41a0422602d02892] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:38:50.279 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:38:50.281 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:38:50.283 INFO 4486 --- [ocp-schedules-29,0f4bdc56b052899e,41a0422602d02892] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:38:50.401 INFO 4486 --- [ocp-schedules-21,0fb3bd6af3640886,cb5d78628ff66d80] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:38:50.401 INFO 4486 --- [ocp-schedules-21,0fb3bd6af3640886,cb5d78628ff66d80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:38:50.407 INFO 4486 --- [ocp-schedules-21,0fb3bd6af3640886,cb5d78628ff66d80] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:38:50.407 INFO 4486 --- [ocp-schedules-21,0fb3bd6af3640886,cb5d78628ff66d80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:38:50.411 INFO 4486 --- [ocp-schedules-21,0fb3bd6af3640886,cb5d78628ff66d80] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:38:50.411 INFO 4486 --- [ocp-schedules-21,0fb3bd6af3640886,cb5d78628ff66d80] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:38:50.484 INFO 4486 --- [ocp-schedules-20,9da2d3903b4e3aa3,69531d90bd1c09f5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:38:51.078 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,98cb9c99826f62da,4a29e82e4a276a7d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=98cb9c99826f62da, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:51.078 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,98cb9c99826f62da,4a29e82e4a276a7d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=98cb9c99826f62da, duration=1 ms] 2024-10-16 13:38:51.101 INFO 4486 --- [ocp-async-2,2c483dd3f6f9d1df,51d385e0ddd9eeef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:51.401 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f5b52528d20f23ef,58f524cfc54a56a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f5b52528d20f23ef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:51.401 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f5b52528d20f23ef,58f524cfc54a56a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f5b52528d20f23ef, duration=1 ms] 2024-10-16 13:38:51.841 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:51.929 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cd1069a77c355235,3740be0e315f31ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=cd1069a77c355235, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:38:51.930 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cd1069a77c355235,3740be0e315f31ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=cd1069a77c355235, duration=2 ms] 2024-10-16 13:38:52.994 INFO 4486 --- [prometheus-minute-schedule-1,e44682f0d9cce697,2475b2057c034243] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:52.995 INFO 4486 --- [ocp-async-5,16767d26ec449256,03fb9e77c96c1a8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:38:53.003 INFO 4486 --- [prometheus-minute-schedule-1,e44682f0d9cce697,2475b2057c034243] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:53.101 INFO 4486 --- [ocp-async-7,de730449b704d8a1,bf1b491c6808d721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:54.477 INFO 4486 --- [ocp-schedules-19,96c08fa44615a897,d2b0a34442097a4b] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:38:54.477+08:00, tenantSize=5 2024-10-16 13:38:54.482 INFO 4486 --- [ocp-schedules-19,96c08fa44615a897,d2b0a34442097a4b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729057134477000 2024-10-16 13:38:54.483 INFO 4486 --- [ocp-schedules-19,96c08fa44615a897,d2b0a34442097a4b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729056834441000 2024-10-16 13:38:54.514 INFO 4486 --- [ocp-schedules-19,96c08fa44615a897,d2b0a34442097a4b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:38:54.520 INFO 4486 --- [ocp-schedules-19,96c08fa44615a897,d2b0a34442097a4b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729057134516000 2024-10-16 13:38:54.520 INFO 4486 --- [ocp-schedules-19,96c08fa44615a897,d2b0a34442097a4b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729056834418000 2024-10-16 13:38:54.536 INFO 4486 --- [ocp-schedules-19,96c08fa44615a897,d2b0a34442097a4b] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:38:54.841 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:55.101 INFO 4486 --- [ocp-async-4,fc00c2b9a578dd10,96269afd51bf598d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:55.470 INFO 4486 --- [ocp-schedules-26,f96427c22ddcca03,17d96455c61642a8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:38:55.808 WARN 4486 --- [scanner-13262508-1,f96427c22ddcca03,aa5f4befdcc017d9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:38:55.839 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ba47b80e2f0e1595,7d48fca7375758a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=ba47b80e2f0e1595, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:38:55.840 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ba47b80e2f0e1595,7d48fca7375758a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=ba47b80e2f0e1595, duration=14 ms] 2024-10-16 13:38:55.943 INFO 4486 --- [alarm-detect-15,f96427c22ddcca03,0e1ad1ffb9562fc3] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057110 ~ 1729057125 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057125, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057125, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057125=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:38:55.994 INFO 4486 --- [prometheus-minute-schedule-1,62e99671d0d8e7cc,fa90dacb87349b5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:56.005 INFO 4486 --- [prometheus-minute-schedule-1,62e99671d0d8e7cc,fa90dacb87349b5b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:57.101 INFO 4486 --- [ocp-async-1,0d01527b055c79db,7ef7eb5c0f85f3e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:57.320 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,289d057a7fa8bc08,acc09a357304bf21] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=289d057a7fa8bc08, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:57.320 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,289d057a7fa8bc08,acc09a357304bf21] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=289d057a7fa8bc08, duration=2 ms] 2024-10-16 13:38:57.842 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:38:58.933 INFO 4486 --- [ocp-schedules-24,456d2d55e2221cc9,44f2e7b09d938a8c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:38:58.938 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cb9e081cd87e7b26,4dcc25a6b291c4b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=cb9e081cd87e7b26, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:38:58.939 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cb9e081cd87e7b26,4dcc25a6b291c4b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=cb9e081cd87e7b26, duration=2 ms] 2024-10-16 13:38:58.994 INFO 4486 --- [prometheus-minute-schedule-1,4d1499c1e0763724,6f5b3a5373dc207b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057140, interval=60 2024-10-16 13:38:59.001 INFO 4486 --- [prometheus-minute-schedule-1,4d1499c1e0763724,6f5b3a5373dc207b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:38:59.102 INFO 4486 --- [ocp-async-10,a45d543cb06702ca,c7ad59afab870a10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:38:59.518 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,89a63a7b069a3f12,faa2ed94e66729aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=89a63a7b069a3f12, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:38:59.518 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,89a63a7b069a3f12,faa2ed94e66729aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=89a63a7b069a3f12, duration=1 ms] 2024-10-16 13:39:00.310 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4634c396c71a6de3,82b947a657ed12e2] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obce_3zones&version=2&ObClusterId=1701313639, client=10.197.49.176, traceId=4634c396c71a6de3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obce_3zones,1701313639,2,{"ObRegion":"obce_3zones","ObCluster":"obce_3zones","ObRegionId":1701313639,"ObClusterId":1701313639,"Type":"PRIMARY","timestamp":1729057140307232,"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:39:00.315 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4634c396c71a6de3,82b947a657ed12e2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=obce_3zones, obClusterId=1701313639, rootServerJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, groupName=obce_3zones, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:39:00.315 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4634c396c71a6de3,82b947a657ed12e2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:39:00.315 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4634c396c71a6de3,82b947a657ed12e2] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=obce_3zones, clusterId=1701313639, affectRows=1 2024-10-16 13:39:00.316 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4634c396c71a6de3,82b947a657ed12e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.176, traceId=4634c396c71a6de3, duration=10 ms] 2024-10-16 13:39:00.489 INFO 4486 --- [ocp-schedules-24,72d756b667b9e0c4,06c826fa1220813f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:39:00.843 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:01.083 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ccc6f5c379099153,abcf39948fa2a039] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ccc6f5c379099153, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:01.083 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ccc6f5c379099153,abcf39948fa2a039] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ccc6f5c379099153, duration=2 ms] 2024-10-16 13:39:01.102 INFO 4486 --- [ocp-async-8,1d659da52e686765,4ffe1dabaf4e8431] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:01.405 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d3cdb511e3a1f075,d1971befb66e5c69] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d3cdb511e3a1f075, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:01.405 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d3cdb511e3a1f075,d1971befb66e5c69] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d3cdb511e3a1f075, duration=1 ms] 2024-10-16 13:39:01.407 INFO 4486 --- [ocp-schedules-27,232c5a4f34d0cb32,ead9bc30e99dea4b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:39:01.407+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:39:01.412 INFO 4486 --- [ocp-schedules-27,232c5a4f34d0cb32,ead9bc30e99dea4b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:39:01.420 INFO 4486 --- [ocp-schedules-27,232c5a4f34d0cb32,ead9bc30e99dea4b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:39:01.431 INFO 4486 --- [ocp-schedules-27,232c5a4f34d0cb32,ead9bc30e99dea4b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:39:01.994 INFO 4486 --- [prometheus-minute-schedule-1,3646bb67b17b0dbc,8db9fc348e080bb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:02.001 INFO 4486 --- [prometheus-minute-schedule-1,3646bb67b17b0dbc,8db9fc348e080bb7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:02.996 INFO 4486 --- [ocp-async-7,760c99b335bd2e71,881b2f4790d9a333] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:39:03.102 INFO 4486 --- [ocp-async-4,f05ec58fb4e1e8ec,b02309962f8e4f93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:03.469 INFO 4486 --- [ocp-schedules-12,f39e2965756d65b3,8845ac4e6b802215] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:39:03.472 INFO 4486 --- [ocp-schedules-12,f39e2965756d65b3,8845ac4e6b802215] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:03.502 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:39:03.503 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:39:03.503 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:39:03.844 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:04.994 INFO 4486 --- [prometheus-minute-schedule-1,fb9d0e31c7e31f2e,c32cd85ccdca3ffc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:05.001 INFO 4486 --- [prometheus-minute-schedule-1,fb9d0e31c7e31f2e,c32cd85ccdca3ffc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:05.102 INFO 4486 --- [ocp-async-2,69f1c2c598b81b2d,55b21c80a40192fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:06.058 INFO 4486 --- [ocp-schedules-1,9c3ed6ab3b9c896d,1fe30676791b5b9f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:39:06.079 WARN 4486 --- [scanner-13262817-1,9c3ed6ab3b9c896d,ea1eec2711ca7d6a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:39:06.845 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:07.066 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:39:07.066 INFO 4486 --- [ocp-schedules-20,24f62054376cc6cb,2ec5dc6ea43c24c5] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:39:07.069 INFO 4486 --- [ocp-schedules-20,24f62054376cc6cb,2ec5dc6ea43c24c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:07.081 INFO 4486 --- [ocp-schedules-6,e36582a852ae1e36,0815e854c5a43b56] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:39:07.083 INFO 4486 --- [ocp-schedules-6,e36582a852ae1e36,0815e854c5a43b56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:39:07.090 INFO 4486 --- [ocp-schedules-5,cb9717f946e909f8,9d440ffccfbad04a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:39:07.090 INFO 4486 --- [ocp-schedules-8,8dec22f24f84d77c,337b76dab8e0aae4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:39:07.091 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:39:07.091 INFO 4486 --- [host-check-clockdiff1,8dec22f24f84d77c,e3a7cdb17910b325] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:39:07.091 INFO 4486 --- [host-check-clockdiff2,8dec22f24f84d77c,d2ddbcad05e936cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:39:07.091 INFO 4486 --- [host-check-clockdiff3,8dec22f24f84d77c,8c2146bcc0f89ba7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:39:07.092 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:39:07.092 INFO 4486 --- [host-check-clockdiff4,8dec22f24f84d77c,9ee1e126e9c7bc61] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:39:07.092 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:39:07.092 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:39:07.092 INFO 4486 --- [ocp-schedules-29,0fd649e5c3ffa0dc,4d341bf5718411ed] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:39:07.093 INFO 4486 --- [host-check-agent-status1,0fd649e5c3ffa0dc,40f107624c84bf15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:39:07.094 INFO 4486 --- [host-check-agent-status2,0fd649e5c3ffa0dc,25e07a50c2659e7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:39:07.094 INFO 4486 --- [host-check-agent-status4,0fd649e5c3ffa0dc,d66982341476ee32] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:39:07.094 INFO 4486 --- [host-check-agent-status3,0fd649e5c3ffa0dc,43b9570fa180e5a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:39:07.105 INFO 4486 --- [host-check-agent-status1,0fd649e5c3ffa0dc,40f107624c84bf15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:07.106 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:39:07.106 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:07.106 INFO 4486 --- [ocp-async-8,f4b80bd6486205df,54c763db7e277820] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:07.146 INFO 4486 --- [host-check-agent-status4,0fd649e5c3ffa0dc,d66982341476ee32] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:39:07.147 INFO 4486 --- [host-check-agent-status3,0fd649e5c3ffa0dc,43b9570fa180e5a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:39:07.147 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:39:07.154 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:39:07.154 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:07.162 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:07.162 INFO 4486 --- [host-check-clockdiff1,8dec22f24f84d77c,e3a7cdb17910b325] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057147 0 0 , error= 2024-10-16 13:39:07.162 INFO 4486 --- [host-check-clockdiff4,8dec22f24f84d77c,9ee1e126e9c7bc61] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057147 3 3 , error= 2024-10-16 13:39:07.162 INFO 4486 --- [host-check-clockdiff3,8dec22f24f84d77c,8c2146bcc0f89ba7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057147 3 3 , error= 2024-10-16 13:39:07.166 INFO 4486 --- [host-check-agent-status4,0fd649e5c3ffa0dc,d66982341476ee32] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:39:07.166 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:39:07.166 INFO 4486 --- [host-check-agent-status3,0fd649e5c3ffa0dc,43b9570fa180e5a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:39:07.166 INFO 4486 --- [host-check-agent-status1,0fd649e5c3ffa0dc,40f107624c84bf15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:07.166 INFO 4486 --- [host-check-clockdiff2,8dec22f24f84d77c,d2ddbcad05e936cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057147 3 3 , error= 2024-10-16 13:39:07.168 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:39:07.170 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:39:07.170 INFO 4486 --- [host-check-agent-status2,0fd649e5c3ffa0dc,25e07a50c2659e7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:39:07.173 INFO 4486 --- [host-check-agent-status2,0fd649e5c3ffa0dc,25e07a50c2659e7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:39:07.176 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.177 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:07.178 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:39:07.187 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:07.189 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.195 INFO 4486 --- [host-check-agent-status3,0fd649e5c3ffa0dc,43b9570fa180e5a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:39:07.195 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:39:07.195 INFO 4486 --- [host-check-agent-status4,0fd649e5c3ffa0dc,d66982341476ee32] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:39:07.195 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:39:07.196 INFO 4486 --- [host-check-agent-status1,0fd649e5c3ffa0dc,40f107624c84bf15] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:39:07.196 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:39:07.196 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.196 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:39:07.197 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:07.198 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:39:07.198171302+08:00 2024-10-16 13:39:07.199 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:39:07.201842715+08:00 2024-10-16 13:39:07.199 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:39:07.201949122+08:00 2024-10-16 13:39:07.199 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:39:07.199+08:00 2024-10-16 13:39:07.200 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:39:07.199+08:00 2024-10-16 13:39:07.200 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:39:07.199+08:00 2024-10-16 13:39:07.201 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:39:07.201 INFO 4486 --- [host-check-agent-status1,0fd649e5c3ffa0dc,40f107624c84bf15] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:39:07.201 INFO 4486 --- [host-check-agent-status1,0fd649e5c3ffa0dc,40f107624c84bf15] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:39:07.201 INFO 4486 --- [host-check-agent-status3,0fd649e5c3ffa0dc,43b9570fa180e5a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:39:07.201 INFO 4486 --- [host-check-agent-status1,0fd649e5c3ffa0dc,40f107624c84bf15] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:39:07.202 INFO 4486 --- [host-check-agent-status3,0fd649e5c3ffa0dc,43b9570fa180e5a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:39:07.202 INFO 4486 --- [host-check-agent-status3,0fd649e5c3ffa0dc,43b9570fa180e5a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:39:07.202 INFO 4486 --- [host-check-agent-status4,0fd649e5c3ffa0dc,d66982341476ee32] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:39:07.202 INFO 4486 --- [host-check-agent-status4,0fd649e5c3ffa0dc,d66982341476ee32] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:39:07.202 INFO 4486 --- [host-check-agent-status4,0fd649e5c3ffa0dc,d66982341476ee32] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:39:07.203 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:39:07.199+08:00 and status AVAILABLE 2024-10-16 13:39:07.203 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:39:07.199+08:00 and status AVAILABLE 2024-10-16 13:39:07.204 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:39:07.199+08:00 and status AVAILABLE 2024-10-16 13:39:07.204 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:39:07.204 INFO 4486 --- [host-check-agent-status2,0fd649e5c3ffa0dc,25e07a50c2659e7d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:39:07.204 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.205 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:39:07.207936596+08:00 2024-10-16 13:39:07.205 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:39:07.206 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:39:07.206 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:39:07.205+08:00 2024-10-16 13:39:07.206 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:39:07.207 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:39:07.209 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:39:07.205+08:00 and status AVAILABLE 2024-10-16 13:39:07.209 INFO 4486 --- [host-check-agent-status2,0fd649e5c3ffa0dc,25e07a50c2659e7d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:39:07.209 INFO 4486 --- [host-check-agent-status2,0fd649e5c3ffa0dc,25e07a50c2659e7d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:39:07.209 INFO 4486 --- [host-check-agent-status2,0fd649e5c3ffa0dc,25e07a50c2659e7d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:39:07.212 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:39:07.216 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:39:07.216 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:39:07.217 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:07.218 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:39:07.218 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:39:07.218 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:07.220 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:39:07.220 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.221 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:39:07.221 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:39:07.238 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.239 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:39:07.242 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.243 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:39:07.244 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:07.244 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:07.244 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:07.245 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:07.252 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:07.253 INFO 4486 --- [ocp-schedules-23,05cc848fdc72c8b2,b54c03492acf873a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:39:07.279 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:39:07.287 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:39:07.293 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:39:07.296 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:39:07.300 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:39:07.302 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:39:07.306 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:39:07.311 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:07.313 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:39:07.316 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:07.324 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2cee678c32ba4727,74951b1caa71b702] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2cee678c32ba4727, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:07.324 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:07.324 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2cee678c32ba4727,74951b1caa71b702] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2cee678c32ba4727, duration=1 ms] 2024-10-16 13:39:07.328 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:07.339 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:39:07.350 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:39:07.352 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:39:07.356 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:39:07.364 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:39:07.367 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:07.369 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:39:07.380 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:07.388 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:39:07.388 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:07.397 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:39:07.399 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:39:07.408 INFO 4486 --- [host-check-host-status3,cb9717f946e909f8,b40e39a7332b4baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:39:07.415 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:07.415 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:39:07.426 INFO 4486 --- [host-check-host-status4,cb9717f946e909f8,c10b1f09ee0d35fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:39:07.429 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:39:07.441 INFO 4486 --- [host-check-host-status2,cb9717f946e909f8,6694432d014eb2d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:39:07.475 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:39:07.488 INFO 4486 --- [host-check-host-status1,cb9717f946e909f8,283aa2b2e14c0fbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:39:07.994 INFO 4486 --- [prometheus-minute-schedule-1,2c14ec9b0a85e14b,5ee081983621cc76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:08.000 INFO 4486 --- [prometheus-minute-schedule-1,2c14ec9b0a85e14b,5ee081983621cc76] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:08.350 INFO 4486 --- [ocp-schedules-16,22c435451135eab3,0e00264ea2db45d7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:39:08.351 INFO 4486 --- [ocp-schedules-16,93ab587a3d2a5153,a37cd023a48fc042] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:39:08.352 INFO 4486 --- [ocp-schedules-16,93ab587a3d2a5153,a37cd023a48fc042] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:08.381 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.382 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:08.387 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.388 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:08.406 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.407 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:08.419 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.420 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.420 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:39:08.449 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.450 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.450 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:08.451 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:39:08.451 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.454 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:39:08.456 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:08.465 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.465 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:08.477 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.486 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.487 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.488 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:08.488 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:39:08.493 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:39:08.497 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:08.499 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:08.516 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.517 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.518 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:39:08.520 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:39:08.555 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:39:08.555 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:39:08.570 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.572 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.572 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:08.576 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.579 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:39:08.583 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:08.591 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:39:08.601 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.601 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.602 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:39:08.609 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.609 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:08.611 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:39:08.611 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:39:08.636 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.636 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:08.661 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.661 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:08.686 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.686 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:08.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:08.739 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:08.766 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.768 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.769 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:08.773 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:39:08.778 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:39:08.782 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:39:08.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:39:08.805 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:39:08.944 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:39:08.975 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:08.977 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:08.978 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:39:09.014 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:39:09.014 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:39:09.106 INFO 4486 --- [ocp-async-3,848242cde0bdf110,e7b58ca69205a3a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:09.525 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,dc507b69cf82f184,90175f989ca2f1e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dc507b69cf82f184, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:09.525 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,dc507b69cf82f184,90175f989ca2f1e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dc507b69cf82f184, duration=2 ms] 2024-10-16 13:39:09.846 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:10.006 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:39:10.006 INFO 4486 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:39:10.025 INFO 4486 --- [Check obproxy1,7816116ffd18db70,15d61531a763b062] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:39:10.025 INFO 4486 --- [Check obproxy1,7816116ffd18db70,15d61531a763b062] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:10.025 INFO 4486 --- [Check obproxy2,bf8e9dc61f3aa3fd,1d80057edcfb2f86] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:39:10.025 INFO 4486 --- [Check obproxy2,bf8e9dc61f3aa3fd,1d80057edcfb2f86] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:10.030 INFO 4486 --- [Collect obproxy1,1a59bdb03c0b2a84,fc814144ea2d95ae] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:39:10.030 INFO 4486 --- [Collect obproxy1,1a59bdb03c0b2a84,fc814144ea2d95ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:10.032 INFO 4486 --- [Collect obproxy2,fea0567739f513e5,a40f8eaa1000efc2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:39:10.032 INFO 4486 --- [Collect obproxy2,fea0567739f513e5,a40f8eaa1000efc2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:10.053 INFO 4486 --- [Check obproxy2,bf8e9dc61f3aa3fd,1d80057edcfb2f86] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:39:10.053 INFO 4486 --- [Check obproxy2,bf8e9dc61f3aa3fd,1d80057edcfb2f86] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:39:10.058 INFO 4486 --- [Collect obproxy1,1a59bdb03c0b2a84,fc814144ea2d95ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:39:10.059 INFO 4486 --- [Collect obproxy2,fea0567739f513e5,a40f8eaa1000efc2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:39:10.061 INFO 4486 --- [Collect obproxy1,1a59bdb03c0b2a84,fc814144ea2d95ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:39:10.064 INFO 4486 --- [Collect obproxy1,1a59bdb03c0b2a84,fc814144ea2d95ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:39:10.065 INFO 4486 --- [Collect obproxy2,fea0567739f513e5,a40f8eaa1000efc2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:39:10.070 INFO 4486 --- [Collect obproxy2,fea0567739f513e5,a40f8eaa1000efc2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:39:10.073 INFO 4486 --- [Check obproxy1,7816116ffd18db70,15d61531a763b062] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:39:10.073 INFO 4486 --- [Check obproxy1,7816116ffd18db70,15d61531a763b062] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:39:10.495 INFO 4486 --- [ocp-schedules-23,d5350c200d36c955,eb1fcb8f0dc997ff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:39:10.980 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:39:10.994 INFO 4486 --- [prometheus-minute-schedule-1,97f523f57d48fcd0,6336fd18474bac2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:11.000 INFO 4486 --- [prometheus-minute-schedule-1,97f523f57d48fcd0,6336fd18474bac2f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:11.008 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:39:11.088 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1392cd6d72c06408,d50f0215170cd1c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1392cd6d72c06408, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:11.089 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1392cd6d72c06408,d50f0215170cd1c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1392cd6d72c06408, duration=1 ms] 2024-10-16 13:39:11.106 INFO 4486 --- [ocp-async-4,380901635a6721fe,edc826b8bc93a075] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:11.410 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cfc9235e666a33a7,16b18a96a14059ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cfc9235e666a33a7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:11.410 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cfc9235e666a33a7,16b18a96a14059ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cfc9235e666a33a7, duration=1 ms] 2024-10-16 13:39:12.847 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:12.997 INFO 4486 --- [ocp-async-4,3986e76ac9c7fadc,2c1cfd3e64c43830] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:39:13.107 INFO 4486 --- [ocp-async-5,ad954f8b2ff47443,a802d7dfb7b569d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:13.463 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057080, nodeCount=1000, affectRows=1000 2024-10-16 13:39:13.467 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057080, nodeCount=999, validCount=999 2024-10-16 13:39:13.468 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057080, nodeCount=366, affectRows=366 2024-10-16 13:39:13.469 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057080, nodeCount=386, affectRows=386 2024-10-16 13:39:13.470 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057080, nodeCount=366, validCount=366 2024-10-16 13:39:13.470 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057080, nodeCount=386, validCount=386 2024-10-16 13:39:13.483 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057080, nodeCount=698, affectRows=698 2024-10-16 13:39:13.485 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057080, nodeCount=696, validCount=696 2024-10-16 13:39:13.487 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057080, nodeCount=1000, affectRows=1000 2024-10-16 13:39:13.490 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057080, nodeCount=1000, validCount=1000 2024-10-16 13:39:13.503 INFO 4486 --- [ocp-schedules-12,f39e2965756d65b3,8845ac4e6b802215] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:39:13.994 INFO 4486 --- [prometheus-minute-schedule-1,2b79418c49fb4d72,ba79f3453fdd48bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:14.000 INFO 4486 --- [prometheus-minute-schedule-1,2b79418c49fb4d72,ba79f3453fdd48bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:14.603 INFO 4486 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057080, nodeCount=66, affectRows=66 2024-10-16 13:39:14.605 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057080, nodeCount=128, affectRows=128 2024-10-16 13:39:14.606 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057080, nodeCount=120, affectRows=120 2024-10-16 13:39:14.609 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057080, nodeCount=166, affectRows=166 2024-10-16 13:39:14.698 INFO 4486 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057080, nodeCount=66, validCount=66 2024-10-16 13:39:14.767 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057080, nodeCount=120, validCount=120 2024-10-16 13:39:14.785 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057080, nodeCount=128, validCount=128 2024-10-16 13:39:14.847 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057080, nodeCount=166, validCount=166 2024-10-16 13:39:15.107 INFO 4486 --- [ocp-async-7,c0be72581a7e56b2,3e08e109141266ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:15.849 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:16.187 INFO 4486 --- [ocp-schedules-8,92b1f7c11a8b1237,16883bd61157da27] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:39:16.230 WARN 4486 --- [scanner-13263857-1,92b1f7c11a8b1237,2d22f1687c9694f6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:39:16.303 INFO 4486 --- [alarm-detect-18,92b1f7c11a8b1237,80744925048585bc] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057125 ~ 1729057146 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057140, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057140, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057140=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:39:16.994 INFO 4486 --- [prometheus-minute-schedule-1,14370a22ba615145,a9a7d18158b464c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:17.000 INFO 4486 --- [prometheus-minute-schedule-1,14370a22ba615145,a9a7d18158b464c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:17.070 INFO 4486 --- [ocp-schedules-20,24f62054376cc6cb,2ec5dc6ea43c24c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:39:17.107 INFO 4486 --- [ocp-async-6,dd54e8f6b52c1686,69d15aa8d0310dab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:17.122 INFO 4486 --- [ocp-schedules-7,bcee6504d864fdc0,bd00a71dd9f84eb1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:39:17.202 INFO 4486 --- [ocp-schedules-6,e36582a852ae1e36,0815e854c5a43b56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:39:17.330 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,626c2c797d692041,7775308da6b2dfce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=626c2c797d692041, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:17.330 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,626c2c797d692041,7775308da6b2dfce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=626c2c797d692041, duration=2 ms] 2024-10-16 13:39:18.850 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:18.937 INFO 4486 --- [ocp-schedules-13,ec659818fc7b99f7,356eb96afe86a675] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:39:19.107 INFO 4486 --- [ocp-async-9,6f7e3d5f8fd16563,d4c7b92bf3ace49e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:19.530 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,837d5739bea9f447,769643d41d417760] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=837d5739bea9f447, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:19.530 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,837d5739bea9f447,769643d41d417760] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=837d5739bea9f447, duration=1 ms] 2024-10-16 13:39:19.994 INFO 4486 --- [prometheus-minute-schedule-1,051237d7c585c85d,f7c9713f3c895ebd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:20.000 INFO 4486 --- [prometheus-minute-schedule-1,051237d7c585c85d,f7c9713f3c895ebd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:20.501 INFO 4486 --- [ocp-schedules-26,ed30a0d12b6a165e,55cdbd020cdc4524] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:39:20.996 INFO 4486 --- [prometheus-cache-monitor1,0f3d6ca3c27b519a,0defad4da6569156] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13859,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:39:20.996 INFO 4486 --- [prometheus-cache-monitor1,0f3d6ca3c27b519a,0defad4da6569156] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:39:21.094 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b30e5e1cc394dd1e,5270071a5085b3b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b30e5e1cc394dd1e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:21.095 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b30e5e1cc394dd1e,5270071a5085b3b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b30e5e1cc394dd1e, duration=3 ms] 2024-10-16 13:39:21.107 INFO 4486 --- [ocp-async-1,31d20c663e76dd75,e5774bfeec72f100] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:21.415 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f2aa9a661f686ac4,db2eaf31f47fc8b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f2aa9a661f686ac4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:21.415 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f2aa9a661f686ac4,db2eaf31f47fc8b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f2aa9a661f686ac4, duration=1 ms] 2024-10-16 13:39:21.850 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:22.936 INFO 4486 --- [ocp-schedules-30,2b93e362c188701f,39fda9de3dd107e4] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:39:22.994 INFO 4486 --- [prometheus-minute-schedule-1,940dfb51ac6ea31d,0f3759f195ea1c0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:22.996 INFO 4486 --- [ocp-async-7,88d936aa459d0cc7,edfd9c74431e8656] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:39:23.002 INFO 4486 --- [prometheus-minute-schedule-1,940dfb51ac6ea31d,0f3759f195ea1c0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:23.108 INFO 4486 --- [ocp-async-10,8d6803bbacae4207,82675e91ef6306d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:24.426 INFO 4486 --- [ocp-schedules-26,45c37ce7e237091c,7fce5db9b34c3188] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:39:24.851 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:24.978 INFO 4486 --- [ocp-schedules-10,da1ba2e5606509f0,6f7b483a0e76f496] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:39:24.978 INFO 4486 --- [ocp-schedules-10,da1ba2e5606509f0,6f7b483a0e76f496] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:39:25.024 INFO 4486 --- [Collect all tenants major compactions1,da1ba2e5606509f0,dd929e050ed0e1c3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:39:25.025 INFO 4486 --- [Collect all tenants major compactions2,da1ba2e5606509f0,f773f8848a355e6d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:39:25.025 INFO 4486 --- [Collect all tenants major compactions3,da1ba2e5606509f0,cfa703983e355d48] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:39:25.026 INFO 4486 --- [Collect all tenants major compactions5,da1ba2e5606509f0,cb02c90df8a1cd49] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:39:25.027 INFO 4486 --- [Collect all tenants major compactions4,da1ba2e5606509f0,77cf98199a5000f3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:39:25.038 INFO 4486 --- [Collect all tenants major compactions1,da1ba2e5606509f0,dd929e050ed0e1c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:25.038 INFO 4486 --- [Collect all tenants major compactions2,da1ba2e5606509f0,f773f8848a355e6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:25.040 INFO 4486 --- [Collect all tenants major compactions1,da1ba2e5606509f0,dd929e050ed0e1c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:39:25.040 INFO 4486 --- [Collect all tenants major compactions2,da1ba2e5606509f0,f773f8848a355e6d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:39:25.043 INFO 4486 --- [Collect all tenants major compactions1,da1ba2e5606509f0,dd929e050ed0e1c3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:39:25.043 INFO 4486 --- [Collect all tenants major compactions2,da1ba2e5606509f0,f773f8848a355e6d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:39:25.044 INFO 4486 --- [Collect all tenants major compactions1,da1ba2e5606509f0,dd929e050ed0e1c3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:39:25.044 INFO 4486 --- [Collect all tenants major compactions2,da1ba2e5606509f0,f773f8848a355e6d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:39:25.046 INFO 4486 --- [Collect all tenants major compactions3,da1ba2e5606509f0,cfa703983e355d48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:25.048 INFO 4486 --- [Collect all tenants major compactions3,da1ba2e5606509f0,cfa703983e355d48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:39:25.053 INFO 4486 --- [Collect all tenants major compactions3,da1ba2e5606509f0,cfa703983e355d48] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:39:25.053 INFO 4486 --- [Collect all tenants major compactions3,da1ba2e5606509f0,cfa703983e355d48] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:39:25.059 INFO 4486 --- [Collect all tenants major compactions5,da1ba2e5606509f0,cb02c90df8a1cd49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:25.060 INFO 4486 --- [Collect all tenants major compactions4,da1ba2e5606509f0,77cf98199a5000f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:25.061 INFO 4486 --- [Collect all tenants major compactions5,da1ba2e5606509f0,cb02c90df8a1cd49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:39:25.061 INFO 4486 --- [Collect all tenants major compactions4,da1ba2e5606509f0,77cf98199a5000f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:39:25.064 INFO 4486 --- [Collect all tenants major compactions4,da1ba2e5606509f0,77cf98199a5000f3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:39:25.065 INFO 4486 --- [Collect all tenants major compactions4,da1ba2e5606509f0,77cf98199a5000f3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:39:25.065 INFO 4486 --- [Collect all tenants major compactions5,da1ba2e5606509f0,cb02c90df8a1cd49] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:39:25.065 INFO 4486 --- [Collect all tenants major compactions5,da1ba2e5606509f0,cb02c90df8a1cd49] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:39:25.067 INFO 4486 --- [ocp-schedules-10,da1ba2e5606509f0,6f7b483a0e76f496] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:39:25.101 INFO 4486 --- [ocp-schedules-14,284720db29433d78,1db4547dbd941256] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:39:25.108 INFO 4486 --- [ocp-async-9,8f3604b640007242,f6649a765db012f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:25.994 INFO 4486 --- [prometheus-minute-schedule-1,d383cd4ded6816b4,54e92da42ed2ade3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:26.001 INFO 4486 --- [prometheus-minute-schedule-1,d383cd4ded6816b4,54e92da42ed2ade3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:26.389 INFO 4486 --- [ocp-schedules-17,9e46ad15916583c0,2382f2a22d9b8c0a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:39:26.435 WARN 4486 --- [scanner-13264196-1,9e46ad15916583c0,cf618f3d3d4aaafd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:39:26.473 INFO 4486 --- [alarm-detect-9,9e46ad15916583c0,3deba5a0d3decbc7] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057140 ~ 1729057156 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057155, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057155, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057155=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:39:27.086 INFO 4486 --- [ocp-schedules-23,e8f489d3b1a22c51,9f7c31d1c1239f11] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:39:27.086+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:39:27.089 INFO 4486 --- [ocp-schedules-23,e8f489d3b1a22c51,9f7c31d1c1239f11] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:39:27.090 INFO 4486 --- [ocp-schedules-23,e8f489d3b1a22c51,9f7c31d1c1239f11] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:39:27.090 INFO 4486 --- [ocp-schedules-23,e8f489d3b1a22c51,9f7c31d1c1239f11] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:39:27.108 INFO 4486 --- [ocp-async-1,8c4d87e5cd8020f9,36cb0a7fce06cb56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:27.341 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,22be693e018fba48,731a7a9086dddb09] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=22be693e018fba48, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:27.341 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,22be693e018fba48,731a7a9086dddb09] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=22be693e018fba48, duration=5 ms] 2024-10-16 13:39:27.852 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:28.994 INFO 4486 --- [prometheus-minute-schedule-1,25e9e091cf60ee44,a125da23d45d8366] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:29.001 INFO 4486 --- [prometheus-minute-schedule-1,25e9e091cf60ee44,a125da23d45d8366] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:29.108 INFO 4486 --- [ocp-async-7,91ff57df60d985e3,3ed39b8cbe714252] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:29.535 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,956cd25c8d50601d,231e70241eed7893] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=956cd25c8d50601d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:29.535 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,956cd25c8d50601d,231e70241eed7893] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=956cd25c8d50601d, duration=1 ms] 2024-10-16 13:39:30.103 INFO 4486 --- [prometheus-second-schedule-1,6cf68191b9f0319e,7e1a1488ada47864] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:39:30.507 INFO 4486 --- [ocp-schedules-20,ddd413154defdd4f,5edefc2c5167ff1f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:39:30.853 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:31.101 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d1def0009140a187,1342ccffc500a056] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d1def0009140a187, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:31.101 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d1def0009140a187,1342ccffc500a056] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d1def0009140a187, duration=2 ms] 2024-10-16 13:39:31.108 INFO 4486 --- [ocp-async-4,29b5da1d5c073ef4,c7ca816a9039653c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:31.419 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e6649840694c65fd,1a1d3638221e3da8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e6649840694c65fd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:31.419 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e6649840694c65fd,1a1d3638221e3da8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e6649840694c65fd, duration=1 ms] 2024-10-16 13:39:31.994 INFO 4486 --- [prometheus-minute-schedule-1,c8927d7556c4d51d,5e6578ab859c5652] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:32.001 INFO 4486 --- [prometheus-minute-schedule-1,c8927d7556c4d51d,5e6578ab859c5652] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:32.997 INFO 4486 --- [ocp-async-2,bbb427780a27ea6c,e473ca83361e54b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:39:33.109 INFO 4486 --- [ocp-async-8,49287042801e62c3,ca3b3fe917061dca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:33.855 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:34.994 INFO 4486 --- [prometheus-minute-schedule-1,753eee7aa6014814,042faae880b09c87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:35.001 INFO 4486 --- [prometheus-minute-schedule-1,753eee7aa6014814,042faae880b09c87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:35.109 INFO 4486 --- [ocp-async-6,956dfc3b623b2a1b,d00b7da1e3c87154] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:35.695 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,46245fbe8a00ee35,4b6c26889de7af9f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=46245fbe8a00ee35, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:39:35.696 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,46245fbe8a00ee35,4b6c26889de7af9f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=46245fbe8a00ee35, duration=2 ms] 2024-10-16 13:39:36.541 INFO 4486 --- [ocp-schedules-12,c3872d99ca4d2fa0,029715e3d5788577] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:39:36.577 WARN 4486 --- [scanner-13264435-1,c3872d99ca4d2fa0,cf1d7dc249c8d28f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:39:36.856 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:37.063 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:39:37.064 INFO 4486 --- [ocp-schedules-9,ddb095637cb34cd4,c1cc7598ef452b5f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:39:37.066 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:39:37.067 INFO 4486 --- [ocp-schedules-7,0cdbed166e88ebf0,f5eed4198379ed03] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:39:37.068 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:39:37.069 INFO 4486 --- [ocp-schedules-7,0cdbed166e88ebf0,f5eed4198379ed03] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:37.071 INFO 4486 --- [ocp-schedules-9,ddb095637cb34cd4,c1cc7598ef452b5f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:39:37.081 INFO 4486 --- [ocp-schedules-28,1f9a9dc13835ddde,2b9f21c1da3b4fc9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:39:37.083 INFO 4486 --- [ocp-schedules-28,1f9a9dc13835ddde,2b9f21c1da3b4fc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:37.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:39:37.089 INFO 4486 --- [ocp-schedules-10,a4dad805ed4ec805,3e4d3a31c67b47eb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:39:37.090 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:39:37.089 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:39:37.089 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:39:37.090 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:39:37.090 INFO 4486 --- [ocp-schedules-4,fefceafa210df58e,b13147cf572bcf45] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:39:37.091 INFO 4486 --- [host-check-clockdiff2,fefceafa210df58e,430810c51a81cf1a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:39:37.091 INFO 4486 --- [host-check-clockdiff1,fefceafa210df58e,af9f206209afa160] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:39:37.092 INFO 4486 --- [host-check-clockdiff3,fefceafa210df58e,f3a0a5099eaeb492] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:39:37.093 INFO 4486 --- [host-check-clockdiff4,fefceafa210df58e,788a746f346c8b8a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:39:37.094 INFO 4486 --- [ocp-schedules-21,0192a13ed9da5b8a,12f05fb4c284835b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:39:37.095 INFO 4486 --- [host-check-agent-status1,0192a13ed9da5b8a,65b144a92b61f4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:39:37.095 INFO 4486 --- [host-check-agent-status2,0192a13ed9da5b8a,ffeaacaa1a65ec7b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:39:37.095 INFO 4486 --- [host-check-agent-status3,0192a13ed9da5b8a,3391fd93a06b4aa6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:39:37.096 INFO 4486 --- [host-check-agent-status4,0192a13ed9da5b8a,9d1a4ade10e77bfa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:39:37.099 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:39:37.099 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:39:37.100 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:37.103 INFO 4486 --- [host-check-agent-status3,0192a13ed9da5b8a,3391fd93a06b4aa6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:39:37.103 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:39:37.103 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:39:37.103 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:37.108 INFO 4486 --- [host-check-agent-status3,0192a13ed9da5b8a,3391fd93a06b4aa6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:39:37.116 INFO 4486 --- [ocp-async-2,0ebd5608334542de,bb2b2f187cb6fe0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:37.122 INFO 4486 --- [host-check-agent-status1,0192a13ed9da5b8a,65b144a92b61f4f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:37.124 INFO 4486 --- [host-check-clockdiff3,fefceafa210df58e,f3a0a5099eaeb492] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057177 3 3 , error= 2024-10-16 13:39:37.130 INFO 4486 --- [host-check-agent-status4,0192a13ed9da5b8a,9d1a4ade10e77bfa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:39:37.132 INFO 4486 --- [host-check-clockdiff1,fefceafa210df58e,af9f206209afa160] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057177 0 0 , error= 2024-10-16 13:39:37.133 INFO 4486 --- [host-check-agent-status2,0192a13ed9da5b8a,ffeaacaa1a65ec7b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:39:37.133 INFO 4486 --- [host-check-agent-status1,0192a13ed9da5b8a,65b144a92b61f4f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:37.133 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:39:37.133 INFO 4486 --- [host-check-agent-status4,0192a13ed9da5b8a,9d1a4ade10e77bfa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:39:37.135 INFO 4486 --- [host-check-agent-status2,0192a13ed9da5b8a,ffeaacaa1a65ec7b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:39:37.135 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:39:37.139 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:37.141 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:39:37.158 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:39:37.160 INFO 4486 --- [host-check-agent-status1,0192a13ed9da5b8a,65b144a92b61f4f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:39:37.160 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:39:37.160 INFO 4486 --- [host-check-agent-status3,0192a13ed9da5b8a,3391fd93a06b4aa6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:39:37.160 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.161 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:37.161 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:37.162 INFO 4486 --- [host-check-agent-status4,0192a13ed9da5b8a,9d1a4ade10e77bfa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:39:37.162 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:39:37.164484405+08:00 2024-10-16 13:39:37.162 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:39:37.162 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:39:37.165319662+08:00 2024-10-16 13:39:37.163 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.163 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:39:37.162+08:00 2024-10-16 13:39:37.163 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:39:37.162+08:00 2024-10-16 13:39:37.164 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:39:37.164 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:39:37.165 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:39:37.164635699+08:00 2024-10-16 13:39:37.165 INFO 4486 --- [host-check-agent-status3,0192a13ed9da5b8a,3391fd93a06b4aa6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:39:37.165 INFO 4486 --- [host-check-agent-status3,0192a13ed9da5b8a,3391fd93a06b4aa6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:39:37.165 INFO 4486 --- [host-check-agent-status3,0192a13ed9da5b8a,3391fd93a06b4aa6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:39:37.165 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:39:37.162+08:00 and status AVAILABLE 2024-10-16 13:39:37.165 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:39:37.162+08:00 and status AVAILABLE 2024-10-16 13:39:37.166 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.166 INFO 4486 --- [host-check-agent-status1,0192a13ed9da5b8a,65b144a92b61f4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:39:37.166 INFO 4486 --- [host-check-agent-status1,0192a13ed9da5b8a,65b144a92b61f4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:39:37.166 INFO 4486 --- [host-check-agent-status1,0192a13ed9da5b8a,65b144a92b61f4f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:39:37.166 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:39:37.166 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:39:37.165+08:00 2024-10-16 13:39:37.166 INFO 4486 --- [host-check-agent-status2,0192a13ed9da5b8a,ffeaacaa1a65ec7b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:39:37.166 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:39:37.169699531+08:00 2024-10-16 13:39:37.167 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:39:37.166+08:00 2024-10-16 13:39:37.167 INFO 4486 --- [host-check-agent-status4,0192a13ed9da5b8a,9d1a4ade10e77bfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:39:37.167 INFO 4486 --- [host-check-agent-status4,0192a13ed9da5b8a,9d1a4ade10e77bfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:39:37.167 INFO 4486 --- [host-check-agent-status4,0192a13ed9da5b8a,9d1a4ade10e77bfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:39:37.168 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.168 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:39:37.168 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:39:37.169 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:39:37.169 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.170 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:39:37.165+08:00 and status AVAILABLE 2024-10-16 13:39:37.170 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:39:37.166+08:00 and status AVAILABLE 2024-10-16 13:39:37.171 INFO 4486 --- [host-check-agent-status2,0192a13ed9da5b8a,ffeaacaa1a65ec7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:39:37.171 INFO 4486 --- [host-check-agent-status2,0192a13ed9da5b8a,ffeaacaa1a65ec7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:39:37.171 INFO 4486 --- [host-check-agent-status2,0192a13ed9da5b8a,ffeaacaa1a65ec7b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:39:37.173 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:39:37.173 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:39:37.174 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:39:37.177 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:39:37.177 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:39:37.178 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:39:37.178 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:39:37.181 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.182 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:39:37.182 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:39:37.183 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:37.183 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:39:37.184 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.184 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.184 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.184 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.184 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.185 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:37.185 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.194 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.195 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:39:37.202 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:39:37.203 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:37.204 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.204 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:37.205 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:39:37.207 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.207 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:37.208 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:39:37.208 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:39:37.219 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.220 INFO 4486 --- [ocp-schedules-3,41c6d3094c671eb1,dfae359bec8270f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:39:37.236 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:39:37.236 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:39:37.240 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:39:37.247 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:39:37.247 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:39:37.253 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:39:37.262 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:37.262 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:37.264 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:39:37.271 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:37.274 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:39:37.291 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:39:37.297 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:39:37.299 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:39:37.301 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:39:37.308 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:39:37.309 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:39:37.320 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:37.321 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:39:37.328 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:37.338 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:37.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,7b1a516e99c7fad2,112d804fe7c44814] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7b1a516e99c7fad2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:37.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,7b1a516e99c7fad2,112d804fe7c44814] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7b1a516e99c7fad2, duration=2 ms] 2024-10-16 13:39:37.348 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:39:37.362 INFO 4486 --- [host-check-host-status3,a4dad805ed4ec805,9e7e57ccf55020f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:39:37.362 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:39:37.365 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:39:37.373 INFO 4486 --- [host-check-host-status4,a4dad805ed4ec805,cba09a0b383cc0b9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:39:37.375 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:39:37.386 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:39:37.391 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:39:37.398 INFO 4486 --- [host-check-host-status2,a4dad805ed4ec805,00be5470545c93d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:39:37.433 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:37.435 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.436 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:39:37.437 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.437 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.438 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:39:37.446 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:39:37.450 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.450 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.450 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.450 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.450 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.450 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.460 INFO 4486 --- [host-check-host-status1,a4dad805ed4ec805,a0c447105d584d3f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:39:37.479 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:39:37.517 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:39:37.560 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:37.575 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:37.576 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.577 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:39:37.578 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:39:37.579 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.579 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:39:37.580 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.581 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.582 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:39:37.592 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.592 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.592 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.592 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.592 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.593 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.600 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:39:37.796 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:39:37.837 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:37.839 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.840 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:39:37.841 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.842 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.843 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:39:37.853 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.853 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.853 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.853 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.853 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:37.853 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:37.873 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:39:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:39:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:39:37.994 INFO 4486 --- [prometheus-minute-schedule-1,40339afc61e6c54a,ab6ca86587987ca1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:38.000 INFO 4486 --- [prometheus-minute-schedule-1,40339afc61e6c54a,ab6ca86587987ca1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:38.053 INFO 4486 --- [host-check-clockdiff4,fefceafa210df58e,788a746f346c8b8a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057178 3 3 , error= 2024-10-16 13:39:38.066 INFO 4486 --- [host-check-clockdiff2,fefceafa210df58e,430810c51a81cf1a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057178 3 3 , error= 2024-10-16 13:39:38.101 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:39:38.143 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:38.145 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:38.146 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:39:38.147 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:38.148 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:38.149 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:39:38.160 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:38.160 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:38.160 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:38.160 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:38.160 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:39:38.161 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:38.223 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:39:38.430 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:39:38.226987, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:39:38.430 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:39:38.226987, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:39:38.469 INFO 4486 --- [ocp-schedules-6,bdd4c359eae609d8,8f0cc4b0a5596034] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13859,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:39:38.469 INFO 4486 --- [ocp-schedules-6,bdd4c359eae609d8,8f0cc4b0a5596034] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:39:38.472 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:39:38.226987} } 2024-10-16 13:39:38.524 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:39:38.529 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:39:38.532 INFO 4486 --- [ocp-schedules-27,fad77ecee78d98fb,416f4ec7e0c9e076] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:39:38.942 INFO 4486 --- [ocp-schedules-1,e6620c2ca5556b9d,34421649a135edd6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:39:39.015 INFO 4486 --- [ocp-schedules-16,93ab587a3d2a5153,a37cd023a48fc042] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:39:39.015 INFO 4486 --- [ocp-schedules-16,05203d33297188ca,622989b6f7b6f69c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:39:39.017 INFO 4486 --- [ocp-schedules-16,05203d33297188ca,622989b6f7b6f69c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:39.042 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.042 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:39.051 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.052 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:39.076 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.077 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:39:39.077 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.078 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:39.107 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.108 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.108 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:39.109 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:39:39.110 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.114 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:39:39.116 INFO 4486 --- [ocp-async-5,6999ae6c5dd7f815,b6d5ae1f8d887694] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:39.116 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:39.125 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.125 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:39.136 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.146 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.146 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.147 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:39.148 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:39:39.152 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:39:39.157 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:39.158 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:39.182 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:39:39.183 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.185 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.185 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:39:39.214 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:39:39.214 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:39:39.235 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.236 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.237 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:39:39.240 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.243 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:39:39.247 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:39.263 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:39:39.274 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.274 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:39.274 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.275 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.276 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:39:39.285 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:39:39.285 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:39:39.301 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.301 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:39.329 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.329 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:39.356 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.356 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:39:39.386 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.386 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:39:39.420 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:39:39.456 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.457 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.458 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:39:39.463 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:39:39.468 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:39:39.473 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:39:39.475 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:39:39.501 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:39:39.541 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c4a8358cb29f9481,50bb08ccef1e2cc6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c4a8358cb29f9481, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:39.541 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c4a8358cb29f9481,50bb08ccef1e2cc6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c4a8358cb29f9481, duration=1 ms] 2024-10-16 13:39:39.645 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:39:39.670 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:39:39.672 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:39.673 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:39:39.699 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:39:39.699 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:39:39.856 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:40.006 INFO 4486 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:39:40.006 INFO 4486 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:39:40.026 INFO 4486 --- [Check obproxy1,aafe292a4918cb14,3caa1a63ea7de911] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:39:40.026 INFO 4486 --- [Check obproxy1,aafe292a4918cb14,3caa1a63ea7de911] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:40.026 INFO 4486 --- [Check obproxy2,00bd7f7d3882fc98,8153fbc00c1a85c7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:39:40.026 INFO 4486 --- [Check obproxy2,00bd7f7d3882fc98,8153fbc00c1a85c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:40.033 INFO 4486 --- [Collect obproxy2,61f6cfd27de299ac,c7c279d51e389dd7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:39:40.033 INFO 4486 --- [Collect obproxy2,61f6cfd27de299ac,c7c279d51e389dd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:40.034 INFO 4486 --- [Collect obproxy1,2e2c47c2ba066111,25eea760534609a4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:39:40.034 INFO 4486 --- [Collect obproxy1,2e2c47c2ba066111,25eea760534609a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:39:40.050 INFO 4486 --- [Check obproxy2,00bd7f7d3882fc98,8153fbc00c1a85c7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:39:40.050 INFO 4486 --- [Check obproxy2,00bd7f7d3882fc98,8153fbc00c1a85c7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:39:40.057 INFO 4486 --- [Check obproxy1,aafe292a4918cb14,3caa1a63ea7de911] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:39:40.057 INFO 4486 --- [Check obproxy1,aafe292a4918cb14,3caa1a63ea7de911] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:39:40.058 INFO 4486 --- [Collect obproxy2,61f6cfd27de299ac,c7c279d51e389dd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:39:40.061 INFO 4486 --- [Collect obproxy1,2e2c47c2ba066111,25eea760534609a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:39:40.064 INFO 4486 --- [Collect obproxy2,61f6cfd27de299ac,c7c279d51e389dd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:39:40.066 INFO 4486 --- [Collect obproxy1,2e2c47c2ba066111,25eea760534609a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:39:40.069 INFO 4486 --- [Collect obproxy2,61f6cfd27de299ac,c7c279d51e389dd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:39:40.071 INFO 4486 --- [Collect obproxy1,2e2c47c2ba066111,25eea760534609a4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:39:40.325 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,16761e841c83647a,363d93cb3908552c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=16761e841c83647a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:39:40.325 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,16761e841c83647a,363d93cb3908552c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=16761e841c83647a, duration=2 ms] 2024-10-16 13:39:40.513 INFO 4486 --- [ocp-schedules-1,c42e68491d62d2c3,eba5f62696e3d390] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:39:40.994 INFO 4486 --- [prometheus-minute-schedule-1,617114c6adbb2180,6649361b9fb0a77c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:40.995 INFO 4486 --- [prometheus-hour-schedule-1,d6d94ecb9cacc8f9,b06ef62417c207d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:39:41.000 INFO 4486 --- [prometheus-minute-schedule-1,617114c6adbb2180,6649361b9fb0a77c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:41.105 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,95694d851198bdc9,76151dba0258d8df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=95694d851198bdc9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:41.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,95694d851198bdc9,76151dba0258d8df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=95694d851198bdc9, duration=1 ms] 2024-10-16 13:39:41.116 INFO 4486 --- [ocp-async-3,d4d63a640df2e701,c0136e4e9990b51e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:41.424 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8ad7582b28b9aa6d,0bcdbfb1edcec603] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8ad7582b28b9aa6d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:41.424 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8ad7582b28b9aa6d,0bcdbfb1edcec603] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8ad7582b28b9aa6d, duration=1 ms] 2024-10-16 13:39:42.102 INFO 4486 --- [ocp-schedules-24,61b660127ba9e64b,5ae1342898aab691] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:39:42.857 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:42.997 INFO 4486 --- [ocp-async-10,bb9fa5a3fe390f15,8fafcd3045e918f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:39:43.116 INFO 4486 --- [ocp-async-4,1c5b1e2bec36c444,815d1f6f1ce02a58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:43.994 INFO 4486 --- [prometheus-minute-schedule-1,f257195248552af6,1f75b73b6896bc09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:44.004 INFO 4486 --- [prometheus-minute-schedule-1,f257195248552af6,1f75b73b6896bc09] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:45.116 INFO 4486 --- [ocp-async-5,4ed45e5d074465dd,07ffb32876a7c703] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:45.476 INFO 4486 --- [ocp-schedules-9,317ab1017c98546d,9abd5310bbfdeacc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:39:45.480 INFO 4486 --- [ocp-schedules-9,317ab1017c98546d,9abd5310bbfdeacc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:39:45.480 INFO 4486 --- [ocp-schedules-9,317ab1017c98546d,9abd5310bbfdeacc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:39:45.496 INFO 4486 --- [ocp-schedules-2,9d78e68da974c5c1,b15dd4347d3fd165] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:39:45.496 INFO 4486 --- [ocp-schedules-2,9d78e68da974c5c1,b15dd4347d3fd165] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:39:45.858 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:46.873 INFO 4486 --- [ocp-schedules-1,5c4f30b49e81b767,fb60bb06abe3dfea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:39:46.898 WARN 4486 --- [scanner-13264794-1,5c4f30b49e81b767,0a07c718b7bc0e53] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:39:46.949 INFO 4486 --- [alarm-detect-5,5c4f30b49e81b767,1e111379d21d3f4c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057155 ~ 1729057176 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057170, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057170, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057170=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:39:46.994 INFO 4486 --- [prometheus-minute-schedule-1,c85921c4736355ab,b6b94bc37ed2010e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:47.002 INFO 4486 --- [prometheus-minute-schedule-1,c85921c4736355ab,b6b94bc37ed2010e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:47.070 INFO 4486 --- [ocp-schedules-7,0cdbed166e88ebf0,f5eed4198379ed03] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:39:47.117 INFO 4486 --- [ocp-async-3,c70caf26f0194253,1c6d7d0d2c72716e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:47.126 INFO 4486 --- [ocp-schedules-18,c07366ae0a7474eb,fabd31596c71ae8d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:39:47.166 INFO 4486 --- [ocp-schedules-28,1f9a9dc13835ddde,2b9f21c1da3b4fc9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:39:47.356 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,06c97512fc8a7b49,6e66a0b1b29388ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=06c97512fc8a7b49, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:47.356 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,06c97512fc8a7b49,6e66a0b1b29388ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=06c97512fc8a7b49, duration=2 ms] 2024-10-16 13:39:48.695 INFO 4486 --- [ocp-schedules-18,43aaca71ed5deac0,df25be28d4513ebe] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:39:48.697 INFO 4486 --- [ocp-schedules-18,43aaca71ed5deac0,df25be28d4513ebe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:39:48.784 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:48.785 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:39:48.786 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:39:48.786 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:39:48.787 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:39:48.859 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:49.117 INFO 4486 --- [ocp-async-9,ac90690f69272ef9,deaf876e35b1217b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:49.545 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7f0aad8abaecd718,840be6bd9cd3bb77] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7f0aad8abaecd718, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:49.546 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7f0aad8abaecd718,840be6bd9cd3bb77] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7f0aad8abaecd718, duration=2 ms] 2024-10-16 13:39:49.994 INFO 4486 --- [prometheus-minute-schedule-1,52e3066905e86529,4fbcecb6c5234458] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:50.000 INFO 4486 --- [prometheus-minute-schedule-1,52e3066905e86529,4fbcecb6c5234458] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:50.291 INFO 4486 --- [ocp-schedules-1,6fa8946951eea1dc,544ff66cd97ad78a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:39:50.291 INFO 4486 --- [ocp-schedules-1,6fa8946951eea1dc,544ff66cd97ad78a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:39:50.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:39:50.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:39:50.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:39:50.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:39:50.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:39:50.299 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:39:50.299 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:39:50.299 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:39:50.299 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:39:50.299 INFO 4486 --- [ocp-schedules-1,6fa8946951eea1dc,544ff66cd97ad78a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:39:50.299 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:39:50.302 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:39:50.302 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:39:50.302 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:39:50.302 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:39:50.302 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:39:50.302 INFO 4486 --- [ocp-schedules-1,6fa8946951eea1dc,544ff66cd97ad78a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:39:50.332 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:39:50.333 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:39:50.334 INFO 4486 --- [ocp-schedules-1,6fa8946951eea1dc,544ff66cd97ad78a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:39:50.334 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:39:50.334 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:39:50.338 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:39:50.344 INFO 4486 --- [ocp-schedules-1,6fa8946951eea1dc,544ff66cd97ad78a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:39:50.415 INFO 4486 --- [ocp-schedules-15,11128eacd88e076b,7b947c02246b9874] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:39:50.415 INFO 4486 --- [ocp-schedules-15,11128eacd88e076b,7b947c02246b9874] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:39:50.416 INFO 4486 --- [ocp-schedules-15,11128eacd88e076b,7b947c02246b9874] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:39:50.416 INFO 4486 --- [ocp-schedules-15,11128eacd88e076b,7b947c02246b9874] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:39:50.417 INFO 4486 --- [ocp-schedules-15,11128eacd88e076b,7b947c02246b9874] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:39:50.417 INFO 4486 --- [ocp-schedules-15,11128eacd88e076b,7b947c02246b9874] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:39:50.519 INFO 4486 --- [ocp-schedules-9,729a419f244ae6b2,c608d2d9202fdc93] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:39:50.623 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:39:50.623 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:39:50.634 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:39:50.634 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:39:50.634 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:39:50.645 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:39:50.645 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:39:50.645 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:39:50.650 INFO 4486 --- [ocp-schedules-30,3fda468bc388cba6,00936624702771b6] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:39:51.113 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b1e15871f3bd1ffa,95de943034a0c8e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b1e15871f3bd1ffa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:51.114 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b1e15871f3bd1ffa,95de943034a0c8e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b1e15871f3bd1ffa, duration=3 ms] 2024-10-16 13:39:51.117 INFO 4486 --- [ocp-async-5,f95445a55e11dfe0,d4675d9cb00b1f25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:51.428 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9e47dcc118f8ebc5,ddad79a052598e34] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9e47dcc118f8ebc5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:51.429 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9e47dcc118f8ebc5,ddad79a052598e34] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9e47dcc118f8ebc5, duration=2 ms] 2024-10-16 13:39:51.860 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:52.248 INFO 4486 --- [ocp-schedules-12,ec0c23014fbab4de,766d4e66b283cb0f] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:39:52.994 INFO 4486 --- [prometheus-minute-schedule-1,49a415b5b9190628,30e43dfb1b422341] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:52.997 INFO 4486 --- [ocp-async-5,8832a79e564173b3,aab5b982ca4330bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:39:53.001 INFO 4486 --- [prometheus-minute-schedule-1,49a415b5b9190628,30e43dfb1b422341] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:53.013 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a7707161e8a8cd83,16462ac2d8b094f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=a7707161e8a8cd83, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:39:53.014 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a7707161e8a8cd83,16462ac2d8b094f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=a7707161e8a8cd83, duration=2 ms] 2024-10-16 13:39:53.117 INFO 4486 --- [ocp-async-6,4a750bd07f129dc4,e8dfcb94ecf22e39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:54.861 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:55.118 INFO 4486 --- [ocp-async-7,a6dba5853faa533e,86ea70d22bf54fa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:55.994 INFO 4486 --- [prometheus-minute-schedule-1,010b78a153589571,7b2de80b51e9d6a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:56.000 INFO 4486 --- [prometheus-minute-schedule-1,010b78a153589571,7b2de80b51e9d6a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:57.004 INFO 4486 --- [ocp-schedules-15,453af319b000e723,382f0227fc7047fb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:39:57.070 WARN 4486 --- [scanner-13265055-1,453af319b000e723,8c7fbd143c13b149] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:39:57.121 INFO 4486 --- [ocp-async-2,a4d233d58155ce07,29f28504a8d7a434] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:57.126 INFO 4486 --- [alarm-detect-12,453af319b000e723,3ec66d06447920b9] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057170 ~ 1729057187 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057185, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057185, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057185=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:39:57.364 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0c01a9e619b597c0,9850710ca8244f17] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0c01a9e619b597c0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:57.364 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0c01a9e619b597c0,9850710ca8244f17] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0c01a9e619b597c0, duration=2 ms] 2024-10-16 13:39:57.862 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:39:58.727 INFO 4486 --- [ocp-schedules-18,43aaca71ed5deac0,df25be28d4513ebe] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:39:58.946 INFO 4486 --- [ocp-schedules-12,6a641b4cd613970b,8a2017271fd126eb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:39:58.994 INFO 4486 --- [prometheus-minute-schedule-1,64c956e253c1a354,1d9c15d2dabb8c86] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057200, interval=60 2024-10-16 13:39:59.002 INFO 4486 --- [prometheus-minute-schedule-1,64c956e253c1a354,1d9c15d2dabb8c86] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:39:59.121 INFO 4486 --- [ocp-async-1,6ddfac6dad9b8f52,98459f0e1dcfada8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:39:59.552 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7eb1ad4dd275959d,5b2c06b457f4add5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7eb1ad4dd275959d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:39:59.552 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7eb1ad4dd275959d,5b2c06b457f4add5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7eb1ad4dd275959d, duration=2 ms] 2024-10-16 13:40:00.410 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:35:00.0, nextRun=2024-10-16 13:40:00.0, currentDate=Wed Oct 16 13:40:00 CST 2024 2024-10-16 13:40:00.410 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:35:00.0, nextRunTime=2024-10-16 13:40:00.0} 2024-10-16 13:40:00.414 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-10-16 13:40:00.414 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.197.49.184', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-10-16T13:40:00.414+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='791e68b70ade5cad', operation=EXECUTE} 2024-10-16 13:40:00.429 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:35:00.0, nextRun=2024-10-16 13:40:00.0, currentDate=Wed Oct 16 13:40:00 CST 2024 2024-10-16 13:40:00.429 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:35:00.0, nextRunTime=2024-10-16 13:40:00.0} 2024-10-16 13:40:00.458 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 1000003] 2024-10-16 13:40:00.473 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.p.s.internal.SqlParseTaskSchedule : The last sql parse service is still running, skip current task 2024-10-16 13:40:00.473 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,430622eb27b85467] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-10-16 13:40:00.526 INFO 4486 --- [ocp-schedules-14,689046f2dfa61fc0,365984ccdb7911ab] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:40:00.863 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:01.119 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,03e6644770bae6de,42bbc577fd2f5409] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=03e6644770bae6de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:01.119 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,03e6644770bae6de,42bbc577fd2f5409] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=03e6644770bae6de, duration=1 ms] 2024-10-16 13:40:01.121 INFO 4486 --- [ocp-async-10,15903f2bd0cd2ea3,8ba5da5746e4847a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:01.438 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7e9d91095e59b576,899c126912c44c29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7e9d91095e59b576, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:01.438 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7e9d91095e59b576,899c126912c44c29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7e9d91095e59b576, duration=1 ms] 2024-10-16 13:40:01.454 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,a2dba43d70336734] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7932845, operation:EXECUTE, state: RUNNING 2024-10-16 13:40:01.465 INFO 4486 --- [ocp-schedules-22,2e34463802204a53,d0f45f56309071d0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:40:01.465+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:40:01.468 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7932845, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266490, task_operation=execute, latest_execution_start_time=2024-10-16T13:40:01.446+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=7932845}, listMap={}}, executor=10.197.49.184 2024-10-16 13:40:01.469 INFO 4486 --- [ocp-schedules-22,2e34463802204a53,d0f45f56309071d0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:40:01.478 INFO 4486 --- [ocp-schedules-22,2e34463802204a53,d0f45f56309071d0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:40:01.484 INFO 4486 --- [ocp-schedules-22,2e34463802204a53,d0f45f56309071d0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster glob_3zone for slowSqlStat, sql count is 0. 2024-10-16 13:40:01.501 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.559 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.685 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.722 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.732 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for OceanBase 2024-10-16 13:40:01.732 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1079.4750314957316], b:[-1.850753682719265E12] 2024-10-16 13:40:01.736 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.801 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.831 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.854 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:40:01.918 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:01.961 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:01.994 INFO 4486 --- [prometheus-minute-schedule-1,7696a9a71ce330d6,73c89a1e3d59985e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:02.000 INFO 4486 --- [prometheus-minute-schedule-1,7696a9a71ce330d6,73c89a1e3d59985e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:02.214 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:02.252 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:02.289 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for obce_3zones 2024-10-16 13:40:02.289 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1295.9606763372929], b:[2.2870925145727866E12] 2024-10-16 13:40:02.293 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:02.424 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:02.456 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:02.488 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:40:02.633 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:02.672 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:02.874 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:02.909 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:02.933 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for glob_3zone 2024-10-16 13:40:02.933 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1292.3504806553328], b:[2.281317553551766E12] 2024-10-16 13:40:02.937 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:02.997 INFO 4486 --- [ocp-async-10,58f8875a1030511f,b12e0019cf10a443] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:40:03.033 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:03.064 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:03.090 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:40:03.137 INFO 4486 --- [ocp-async-4,e70c2786d132ebc2,3e5a57a452a3f9f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:03.216 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,bc9f5bb48ae1109a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7932845, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 13:40:03.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,31459529cdfb71da,5221f7b73eeac96c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=31459529cdfb71da, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 13:40:03.566 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,31459529cdfb71da,5221f7b73eeac96c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=31459529cdfb71da, duration=225 ms] 2024-10-16 13:40:03.864 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:04.581 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,f254e7b6dcbbbf60] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:30:05.0, nextRun=2024-10-16 13:40:04.0, currentDate=Wed Oct 16 13:40:04 CST 2024 2024-10-16 13:40:04.581 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,f254e7b6dcbbbf60] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-10-16 13:30:05.0, nextRunTime=2024-10-16 13:40:04.0} 2024-10-16 13:40:04.585 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,f254e7b6dcbbbf60] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-10-16 13:40:04.585 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,f254e7b6dcbbbf60] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.197.49.184', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-10-16T13:40:04.585+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='791e68b70ade5cad', operation=EXECUTE} 2024-10-16 13:40:04.994 INFO 4486 --- [prometheus-minute-schedule-1,68d45253f6b01bbb,cc5b978b64fdb2b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:05.000 INFO 4486 --- [prometheus-minute-schedule-1,68d45253f6b01bbb,cc5b978b64fdb2b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:05.137 INFO 4486 --- [ocp-async-5,03376c21ffe1ce9c,adfe80eff79a2cd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:06.865 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:07.066 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:40:07.067 INFO 4486 --- [ocp-schedules-6,9eb441541b22073b,122db7e11d339366] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:40:07.069 INFO 4486 --- [ocp-schedules-6,9eb441541b22073b,122db7e11d339366] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:07.081 INFO 4486 --- [ocp-schedules-7,237c86690577c7b8,97233d724e82223b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:40:07.083 INFO 4486 --- [ocp-schedules-7,237c86690577c7b8,97233d724e82223b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:40:07.090 INFO 4486 --- [ocp-schedules-21,d6ec94327ce6d5ab,9aa4e5a30a2db135] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:40:07.090 INFO 4486 --- [ocp-schedules-1,ab4f081b5245645a,eb20454dfa2149c4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:40:07.091 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:40:07.091 INFO 4486 --- [host-check-clockdiff1,d6ec94327ce6d5ab,6984b42fdfd4d641] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:40:07.091 INFO 4486 --- [host-check-clockdiff2,d6ec94327ce6d5ab,96123a54a6cbda57] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:40:07.091 INFO 4486 --- [host-check-clockdiff3,d6ec94327ce6d5ab,f79bfcadb47fe7ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:40:07.091 INFO 4486 --- [ocp-schedules-3,42d89ab628eaf1ad,09e68a2be8cf32e3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:40:07.091 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:40:07.091 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:40:07.091 INFO 4486 --- [host-check-clockdiff4,d6ec94327ce6d5ab,5da6a413546fd595] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:40:07.092 INFO 4486 --- [host-check-agent-status1,42d89ab628eaf1ad,1b0a474879e26ada] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:40:07.092 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:40:07.093 INFO 4486 --- [host-check-agent-status2,42d89ab628eaf1ad,834bf7b1e9131ca4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:40:07.093 INFO 4486 --- [host-check-agent-status4,42d89ab628eaf1ad,5f29fb8232be7897] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:40:07.094 INFO 4486 --- [host-check-agent-status3,42d89ab628eaf1ad,5eea2428ca7d23b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:40:07.098 INFO 4486 --- [host-check-agent-status1,42d89ab628eaf1ad,1b0a474879e26ada] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:07.099 INFO 4486 --- [host-check-agent-status2,42d89ab628eaf1ad,834bf7b1e9131ca4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:40:07.099 INFO 4486 --- [host-check-agent-status3,42d89ab628eaf1ad,5eea2428ca7d23b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:40:07.099 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:07.100 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:40:07.100 INFO 4486 --- [host-check-agent-status4,42d89ab628eaf1ad,5f29fb8232be7897] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:40:07.101 INFO 4486 --- [host-check-agent-status2,42d89ab628eaf1ad,834bf7b1e9131ca4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:40:07.101 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:07.101 INFO 4486 --- [host-check-agent-status3,42d89ab628eaf1ad,5eea2428ca7d23b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:40:07.101 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:40:07.101 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:40:07.101 INFO 4486 --- [host-check-agent-status4,42d89ab628eaf1ad,5f29fb8232be7897] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:40:07.102 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:40:07.101 INFO 4486 --- [host-check-agent-status1,42d89ab628eaf1ad,1b0a474879e26ada] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:07.104 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:40:07.104 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:40:07.107 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.108 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:07.109 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:40:07.129 INFO 4486 --- [host-check-clockdiff3,d6ec94327ce6d5ab,f79bfcadb47fe7ab] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057207 3 3 , error= 2024-10-16 13:40:07.129 INFO 4486 --- [host-check-clockdiff1,d6ec94327ce6d5ab,6984b42fdfd4d641] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057207 0 0 , error= 2024-10-16 13:40:07.132 INFO 4486 --- [host-check-clockdiff2,d6ec94327ce6d5ab,96123a54a6cbda57] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057207 3 3 , error= 2024-10-16 13:40:07.138 INFO 4486 --- [ocp-async-10,96c4626bd40b55cf,784a73af88922ea8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:07.154 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.156 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:07.156 INFO 4486 --- [host-check-agent-status2,42d89ab628eaf1ad,834bf7b1e9131ca4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:40:07.157 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:40:07.157 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:40:07.158 INFO 4486 --- [host-check-agent-status1,42d89ab628eaf1ad,1b0a474879e26ada] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:40:07.158 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:40:07.159 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:40:07.159 INFO 4486 --- [host-check-agent-status3,42d89ab628eaf1ad,5eea2428ca7d23b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:40:07.159 INFO 4486 --- [host-check-agent-status4,42d89ab628eaf1ad,5f29fb8232be7897] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:40:07.160 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:40:07.159837316+08:00 2024-10-16 13:40:07.160 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:40:07.163300985+08:00 2024-10-16 13:40:07.160 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:40:07.163426525+08:00 2024-10-16 13:40:07.161 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:40:07.160+08:00 2024-10-16 13:40:07.161 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:40:07.160+08:00 2024-10-16 13:40:07.161 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:40:07.161 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:40:07.160+08:00 2024-10-16 13:40:07.161 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:40:07.163590574+08:00 2024-10-16 13:40:07.161 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:40:07.161+08:00 2024-10-16 13:40:07.163 INFO 4486 --- [host-check-agent-status1,42d89ab628eaf1ad,1b0a474879e26ada] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:40:07.163 INFO 4486 --- [host-check-agent-status1,42d89ab628eaf1ad,1b0a474879e26ada] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:40:07.163 INFO 4486 --- [host-check-agent-status1,42d89ab628eaf1ad,1b0a474879e26ada] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:40:07.163 INFO 4486 --- [host-check-agent-status3,42d89ab628eaf1ad,5eea2428ca7d23b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:40:07.163 INFO 4486 --- [host-check-agent-status3,42d89ab628eaf1ad,5eea2428ca7d23b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:40:07.163 INFO 4486 --- [host-check-agent-status3,42d89ab628eaf1ad,5eea2428ca7d23b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:40:07.164 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:40:07.160+08:00 and status AVAILABLE 2024-10-16 13:40:07.164 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:40:07.160+08:00 and status AVAILABLE 2024-10-16 13:40:07.164 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:40:07.160+08:00 and status AVAILABLE 2024-10-16 13:40:07.164 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:40:07.161+08:00 and status AVAILABLE 2024-10-16 13:40:07.164 INFO 4486 --- [host-check-agent-status2,42d89ab628eaf1ad,834bf7b1e9131ca4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:40:07.165 INFO 4486 --- [host-check-agent-status2,42d89ab628eaf1ad,834bf7b1e9131ca4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:40:07.165 INFO 4486 --- [host-check-agent-status2,42d89ab628eaf1ad,834bf7b1e9131ca4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:40:07.165 INFO 4486 --- [host-check-agent-status4,42d89ab628eaf1ad,5f29fb8232be7897] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:40:07.165 INFO 4486 --- [host-check-agent-status4,42d89ab628eaf1ad,5f29fb8232be7897] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:40:07.165 INFO 4486 --- [host-check-agent-status4,42d89ab628eaf1ad,5f29fb8232be7897] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:40:07.166 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:40:07.166 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:40:07.166 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:40:07.166 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:40:07.168 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.173 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:40:07.174 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:40:07.175 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:40:07.176 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:40:07.176 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:07.176 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:40:07.177 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:40:07.177 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:07.178 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:40:07.183 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.184 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:07.194 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.195 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:07.197 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:07.199 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:07.200 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:07.201 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:07.204 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.205 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:07.208 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.209 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:40:07.222 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:07.223 INFO 4486 --- [ocp-schedules-2,ef8407e4e674c631,efe71284209fa23e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:40:07.229 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:40:07.229 INFO 4486 --- [ocp-schedules-15,860939b0d2aa0a70,1645d1b57ac59d50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:40:07.240 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:40:07.241 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:40:07.242 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:40:07.258 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:40:07.263 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:40:07.272 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:40:07.279 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:40:07.283 WARN 4486 --- [scanner-13265661-1,860939b0d2aa0a70,dac0780f728a49b1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:40:07.291 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:07.299 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:07.299 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:07.302 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:07.324 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:40:07.332 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:40:07.342 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:40:07.349 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:40:07.357 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:40:07.357 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:40:07.367 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:40:07.374 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d1523d03dfa242e7,832aab1a152f2229] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d1523d03dfa242e7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:07.375 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:40:07.376 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d1523d03dfa242e7,832aab1a152f2229] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d1523d03dfa242e7, duration=6 ms] 2024-10-16 13:40:07.389 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:07.389 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:07.401 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:07.415 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:07.419 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:40:07.424 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:40:07.431 INFO 4486 --- [host-check-host-status3,ab4f081b5245645a,80289e817adeea0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:40:07.434 INFO 4486 --- [host-check-host-status4,ab4f081b5245645a,01efe98f38bca3c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:40:07.440 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:40:07.450 INFO 4486 --- [host-check-host-status2,ab4f081b5245645a,3a1bcbbcadb49ff6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:40:07.471 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:40:07.485 INFO 4486 --- [host-check-host-status1,ab4f081b5245645a,837b6db0c8f2e397] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:40:07.994 INFO 4486 --- [prometheus-minute-schedule-1,bb2cc98bd18dca22,19de3e79c2b82b9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:08.001 INFO 4486 --- [prometheus-minute-schedule-1,bb2cc98bd18dca22,19de3e79c2b82b9f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:08.060 INFO 4486 --- [host-check-clockdiff4,d6ec94327ce6d5ab,5da6a413546fd595] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057208 3 3 , error= 2024-10-16 13:40:09.138 INFO 4486 --- [ocp-async-7,cdd9a92155882d05,ed6b5cb91ec84eaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:09.557 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,61c60ae5fc18d5e7,568f678a53d64309] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=61c60ae5fc18d5e7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:09.557 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,61c60ae5fc18d5e7,568f678a53d64309] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=61c60ae5fc18d5e7, duration=1 ms] 2024-10-16 13:40:09.700 INFO 4486 --- [ocp-schedules-16,05203d33297188ca,622989b6f7b6f69c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:40:09.700 INFO 4486 --- [ocp-schedules-16,d178e31092876514,424de0a04f0a051a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:40:09.702 INFO 4486 --- [ocp-schedules-16,d178e31092876514,424de0a04f0a051a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:09.730 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.731 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:09.743 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.744 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:09.768 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.768 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:09.772 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:09.773 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.774 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:40:09.802 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:09.802 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.803 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:09.804 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:40:09.804 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.807 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:40:09.808 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:09.818 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:09.818 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:09.828 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:09.837 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:09.837 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.838 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:09.838 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:40:09.843 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:40:09.847 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:09.849 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:09.866 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:09.872 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:40:09.925 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:09.927 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.928 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:40:09.949 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:40:09.953 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:40:09.953 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:40:09.960 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:09.960 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.961 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:40:09.970 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:40:09.970 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:40:09.977 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:09.978 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.979 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:09.982 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:09.986 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:40:09.990 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:10.006 INFO 4486 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:40:10.006 INFO 4486 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:40:10.016 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:10.017 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:10.027 INFO 4486 --- [Check obproxy2,6a3e72d346e86718,746e45e7635c6743] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:40:10.028 INFO 4486 --- [Check obproxy2,6a3e72d346e86718,746e45e7635c6743] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:10.028 INFO 4486 --- [Check obproxy1,9dbbb34276977b23,9e3446cdd145ef68] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:40:10.028 INFO 4486 --- [Check obproxy1,9dbbb34276977b23,9e3446cdd145ef68] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:10.030 INFO 4486 --- [Collect obproxy1,68238c66e9b51a41,51a5a9e71d584d7c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:40:10.030 INFO 4486 --- [Collect obproxy1,68238c66e9b51a41,51a5a9e71d584d7c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:10.033 INFO 4486 --- [Collect obproxy2,9c3d739ca03a12aa,5827b8db7b58a3ba] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:40:10.033 INFO 4486 --- [Collect obproxy2,9c3d739ca03a12aa,5827b8db7b58a3ba] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:10.044 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:10.044 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:10.050 INFO 4486 --- [Check obproxy2,6a3e72d346e86718,746e45e7635c6743] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:40:10.050 INFO 4486 --- [Check obproxy2,6a3e72d346e86718,746e45e7635c6743] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:40:10.051 INFO 4486 --- [Check obproxy1,9dbbb34276977b23,9e3446cdd145ef68] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:40:10.052 INFO 4486 --- [Check obproxy1,9dbbb34276977b23,9e3446cdd145ef68] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:40:10.054 INFO 4486 --- [Collect obproxy2,9c3d739ca03a12aa,5827b8db7b58a3ba] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:40:10.055 INFO 4486 --- [Collect obproxy1,68238c66e9b51a41,51a5a9e71d584d7c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:40:10.058 INFO 4486 --- [Collect obproxy1,68238c66e9b51a41,51a5a9e71d584d7c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:40:10.058 INFO 4486 --- [Collect obproxy2,9c3d739ca03a12aa,5827b8db7b58a3ba] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:40:10.062 INFO 4486 --- [Collect obproxy1,68238c66e9b51a41,51a5a9e71d584d7c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:40:10.063 INFO 4486 --- [Collect obproxy2,9c3d739ca03a12aa,5827b8db7b58a3ba] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:40:10.070 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:10.070 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:10.097 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:10.097 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:10.123 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:10.123 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:10.151 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:10.176 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:10.177 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:10.178 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:10.182 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:40:10.187 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:40:10.192 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:40:10.193 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:40:10.221 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:40:10.357 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:40:10.386 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:10.387 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:10.388 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:40:10.417 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:40:10.417 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:40:10.532 INFO 4486 --- [ocp-schedules-19,0b41613beb96dc17,0aa532fb4ac6b481] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:40:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:40:10.994 INFO 4486 --- [prometheus-minute-schedule-1,49c3018c1f0d0212,401a7f5ddac22b18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:11.001 INFO 4486 --- [prometheus-minute-schedule-1,49c3018c1f0d0212,401a7f5ddac22b18] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:40:11.122 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,09e52e77a5f9fbed,fa35d6a554f871ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=09e52e77a5f9fbed, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:11.123 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,09e52e77a5f9fbed,fa35d6a554f871ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=09e52e77a5f9fbed, duration=1 ms] 2024-10-16 13:40:11.138 INFO 4486 --- [ocp-async-3,df6b2ebf23811196,e45990376201e751] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:11.444 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0364f77bf68a84d7,9ce823537f8fa57f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0364f77bf68a84d7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:11.444 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0364f77bf68a84d7,9ce823537f8fa57f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0364f77bf68a84d7, duration=1 ms] 2024-10-16 13:40:12.513 INFO 4486 --- [ocp-schedules-8,b87ced541b4aac55,216598bc35c853f7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 13:40:12.513 INFO 4486 --- [ocp-schedules-8,b87ced541b4aac55,216598bc35c853f7] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 13:40:12.547 INFO 4486 --- [pool-279-thread-3,b87ced541b4aac55,b9771661011ff63b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:40:12.547 INFO 4486 --- [pool-279-thread-1,b87ced541b4aac55,e6905ff6e424da82] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:40:12.548 INFO 4486 --- [pool-279-thread-2,b87ced541b4aac55,67fc2ace5da4fd41] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:40:12.566 INFO 4486 --- [pool-279-thread-1,b87ced541b4aac55,e6905ff6e424da82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:12.570 INFO 4486 --- [pool-279-thread-2,b87ced541b4aac55,67fc2ace5da4fd41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:12.570 INFO 4486 --- [pool-279-thread-1,b87ced541b4aac55,e6905ff6e424da82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057212000000] 2024-10-16 13:40:12.577 INFO 4486 --- [pool-279-thread-3,b87ced541b4aac55,b9771661011ff63b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:12.579 INFO 4486 --- [pool-279-thread-2,b87ced541b4aac55,67fc2ace5da4fd41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057212000000] 2024-10-16 13:40:12.581 INFO 4486 --- [pool-279-thread-3,b87ced541b4aac55,b9771661011ff63b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057212000000] 2024-10-16 13:40:12.853 WARN 4486 --- [pool-279-thread-3,b87ced541b4aac55,b9771661011ff63b] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 13:40:12.853 WARN 4486 --- [pool-279-thread-3,b87ced541b4aac55,b9771661011ff63b] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 13:40:12.855 INFO 4486 --- [pool-279-thread-3,b87ced541b4aac55,b9771661011ff63b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:40:12.867 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:12.998 INFO 4486 --- [ocp-async-1,4b780b5dc9881493,7a76c39f13fa9a73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:40:13.139 INFO 4486 --- [ocp-async-9,bc179330846c3301,10a522d8b6233e11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:13.203 WARN 4486 --- [pool-279-thread-1,b87ced541b4aac55,e6905ff6e424da82] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 13:40:13.203 WARN 4486 --- [pool-279-thread-1,b87ced541b4aac55,e6905ff6e424da82] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1009 2024-10-16 13:40:13.205 INFO 4486 --- [pool-279-thread-1,b87ced541b4aac55,e6905ff6e424da82] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:40:13.378 WARN 4486 --- [pool-279-thread-2,b87ced541b4aac55,67fc2ace5da4fd41] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 13:40:13.380 INFO 4486 --- [pool-279-thread-2,b87ced541b4aac55,67fc2ace5da4fd41] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:40:13.382 INFO 4486 --- [ocp-schedules-8,b87ced541b4aac55,216598bc35c853f7] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 13:40:13.504 INFO 4486 --- [ocp-schedules-9,add941377e29fc8c,348fe4d87540def6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:40:13.506 INFO 4486 --- [ocp-schedules-9,add941377e29fc8c,348fe4d87540def6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:13.507 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:40:13.507 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:40:13.507 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:40:13.518 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057140, nodeCount=450, affectRows=450 2024-10-16 13:40:13.521 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057140, nodeCount=448, validCount=448 2024-10-16 13:40:13.529 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057140, nodeCount=1000, affectRows=1000 2024-10-16 13:40:13.534 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057140, nodeCount=1000, affectRows=1000 2024-10-16 13:40:13.535 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057140, nodeCount=1000, validCount=1000 2024-10-16 13:40:13.537 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057140, nodeCount=1000, validCount=1000 2024-10-16 13:40:13.541 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057140, nodeCount=1000, affectRows=1000 2024-10-16 13:40:13.544 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057140, nodeCount=999, validCount=999 2024-10-16 13:40:13.994 INFO 4486 --- [prometheus-minute-schedule-1,326531519c42d511,450cce8fecf0d283] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:14.000 INFO 4486 --- [prometheus-minute-schedule-1,326531519c42d511,450cce8fecf0d283] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:14.635 INFO 4486 --- [metric-archive-async-12,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057140, nodeCount=215, affectRows=215 2024-10-16 13:40:14.648 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057140, nodeCount=262, affectRows=262 2024-10-16 13:40:14.999 INFO 4486 --- [metric-archive-async-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057140, nodeCount=215, validCount=215 2024-10-16 13:40:15.065 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057140, nodeCount=262, validCount=262 2024-10-16 13:40:15.139 INFO 4486 --- [ocp-async-7,a0daf464a4c09da6,9742ec4b0d29586c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:15.868 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:16.994 INFO 4486 --- [prometheus-minute-schedule-1,dd14ca708b03b10d,9f9d1c9ca0dc77ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:17.002 INFO 4486 --- [prometheus-minute-schedule-1,dd14ca708b03b10d,9f9d1c9ca0dc77ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:17.071 INFO 4486 --- [ocp-schedules-6,9eb441541b22073b,122db7e11d339366] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:40:17.130 INFO 4486 --- [ocp-schedules-3,c5e8e83a7234e0e3,d09b18a084751ac2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:40:17.139 INFO 4486 --- [ocp-async-4,e5025e07f4776b24,72b06402e5f4984c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:17.161 INFO 4486 --- [ocp-schedules-7,237c86690577c7b8,97233d724e82223b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:40:17.394 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b2cb53a6b53f6bd9,c1c8f4feb33e06b3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b2cb53a6b53f6bd9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:17.394 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b2cb53a6b53f6bd9,c1c8f4feb33e06b3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b2cb53a6b53f6bd9, duration=2 ms] 2024-10-16 13:40:17.412 INFO 4486 --- [ocp-schedules-11,6e5fd13bf43f7d6e,1e3dfefbf5de883d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:40:17.450 WARN 4486 --- [scanner-13266700-1,6e5fd13bf43f7d6e,638a70d8ec5b9324] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:40:17.511 INFO 4486 --- [alarm-detect-8,6e5fd13bf43f7d6e,64f3f69edef10035] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057185 ~ 1729057207 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057200, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057200, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057200=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:40:18.869 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:18.951 INFO 4486 --- [ocp-schedules-3,1233821beb36a20a,e575ab8e1c810e09] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:40:19.139 INFO 4486 --- [ocp-async-5,d358b17ebf4e3a41,555898ffa424061c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:19.483 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0c6810e40953eca1,c20fd89f8f9a0f79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=0c6810e40953eca1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:40:19.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0c6810e40953eca1,c20fd89f8f9a0f79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=0c6810e40953eca1, duration=2 ms] 2024-10-16 13:40:19.567 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,35635e9df47f527a,60fc164ce88007de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=35635e9df47f527a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:19.567 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,35635e9df47f527a,60fc164ce88007de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=35635e9df47f527a, duration=1 ms] 2024-10-16 13:40:19.994 INFO 4486 --- [prometheus-minute-schedule-1,7f9a5070f33d9913,a6c9f464d24183d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:20.000 INFO 4486 --- [prometheus-minute-schedule-1,7f9a5070f33d9913,a6c9f464d24183d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:20.537 INFO 4486 --- [ocp-schedules-23,be94d84c876fbd66,aa4b5fa4222a89e2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:40:20.811 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:40:20.996 INFO 4486 --- [prometheus-cache-monitor1,8441d37cd8ea6cd7,5ccfa7c5b328ec2b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13859,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:40:20.996 INFO 4486 --- [prometheus-cache-monitor1,8441d37cd8ea6cd7,5ccfa7c5b328ec2b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:40:21.127 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5745493f05e0e58b,658cb0818cb25097] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5745493f05e0e58b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:21.127 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5745493f05e0e58b,658cb0818cb25097] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5745493f05e0e58b, duration=1 ms] 2024-10-16 13:40:21.139 INFO 4486 --- [ocp-async-10,30bcb1360cce211a,8382abcddafaef7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:21.449 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,307f6f34f80b1117,fa5eba751d882311] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=307f6f34f80b1117, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:21.449 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,307f6f34f80b1117,fa5eba751d882311] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=307f6f34f80b1117, duration=1 ms] 2024-10-16 13:40:21.711 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d73868e6426b0b40,77cd808a739dec2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=d73868e6426b0b40, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:40:21.712 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d73868e6426b0b40,77cd808a739dec2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=d73868e6426b0b40, duration=1 ms] 2024-10-16 13:40:21.870 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:22.973 INFO 4486 --- [ocp-schedules-26,4b8db0daed5eb3d5,5e67f3856d0f2508] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:40:22.994 INFO 4486 --- [prometheus-minute-schedule-1,4640e1fa2a67047d,0d16557fe237e5b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:22.998 INFO 4486 --- [ocp-async-10,74ca0d4b5610c5cd,0bf54d8c5213f539] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:40:23.001 INFO 4486 --- [prometheus-minute-schedule-1,4640e1fa2a67047d,0d16557fe237e5b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:23.140 INFO 4486 --- [ocp-async-3,36825dac90437d78,e0f530607e88c6f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:23.507 INFO 4486 --- [ocp-schedules-9,add941377e29fc8c,348fe4d87540def6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:40:24.458 INFO 4486 --- [ocp-schedules-13,d911ef0def5b8942,474977b27bbcd12d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:40:24.871 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:25.067 INFO 4486 --- [ocp-schedules-30,3589673fc55cac40,07462608b6db593f] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:40:25.067 INFO 4486 --- [ocp-schedules-30,3589673fc55cac40,07462608b6db593f] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:40:25.102 INFO 4486 --- [ocp-schedules-10,8e9020d2a3dc2666,425b4bc3df08c1f2] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:40:25.103 INFO 4486 --- [Collect all tenants major compactions1,3589673fc55cac40,026882a204aff963] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:40:25.103 INFO 4486 --- [ocp-schedules-10,8e9020d2a3dc2666,425b4bc3df08c1f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:25.103 INFO 4486 --- [Collect all tenants major compactions4,3589673fc55cac40,a46ba1f4646eebfc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:40:25.103 INFO 4486 --- [Collect all tenants major compactions2,3589673fc55cac40,f432e5f0d05e9a7e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:40:25.104 INFO 4486 --- [Collect all tenants major compactions5,3589673fc55cac40,3606428a5ba37f67] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:40:25.104 INFO 4486 --- [Collect all tenants major compactions3,3589673fc55cac40,bcc146e206bec853] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:40:25.116 INFO 4486 --- [Collect all tenants major compactions1,3589673fc55cac40,026882a204aff963] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.116 INFO 4486 --- [Collect all tenants major compactions2,3589673fc55cac40,f432e5f0d05e9a7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.117 INFO 4486 --- [Collect all tenants major compactions1,3589673fc55cac40,026882a204aff963] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:40:25.118 INFO 4486 --- [Collect all tenants major compactions2,3589673fc55cac40,f432e5f0d05e9a7e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:40:25.121 INFO 4486 --- [Collect all tenants major compactions1,3589673fc55cac40,026882a204aff963] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:40:25.121 INFO 4486 --- [Collect all tenants major compactions2,3589673fc55cac40,f432e5f0d05e9a7e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:40:25.121 INFO 4486 --- [Collect all tenants major compactions2,3589673fc55cac40,f432e5f0d05e9a7e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:40:25.121 INFO 4486 --- [Collect all tenants major compactions1,3589673fc55cac40,026882a204aff963] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:40:25.123 INFO 4486 --- [Collect all tenants major compactions3,3589673fc55cac40,bcc146e206bec853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.124 INFO 4486 --- [Collect all tenants major compactions3,3589673fc55cac40,bcc146e206bec853] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:40:25.130 INFO 4486 --- [Collect all tenants major compactions3,3589673fc55cac40,bcc146e206bec853] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:40:25.130 INFO 4486 --- [Collect all tenants major compactions3,3589673fc55cac40,bcc146e206bec853] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:40:25.137 INFO 4486 --- [Collect all tenants major compactions4,3589673fc55cac40,a46ba1f4646eebfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.138 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:40:25.138 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.139 INFO 4486 --- [Collect all tenants major compactions5,3589673fc55cac40,3606428a5ba37f67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.140 INFO 4486 --- [Collect all tenants major compactions4,3589673fc55cac40,a46ba1f4646eebfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:40:25.140 INFO 4486 --- [ocp-async-9,1de96f7c1e51e703,d04a4ed95ca6110e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:25.141 INFO 4486 --- [Collect all tenants major compactions5,3589673fc55cac40,3606428a5ba37f67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:40:25.144 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:40:25.145 INFO 4486 --- [Collect all tenants major compactions4,3589673fc55cac40,a46ba1f4646eebfc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:40:25.145 INFO 4486 --- [Collect all tenants major compactions4,3589673fc55cac40,a46ba1f4646eebfc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:40:25.147 INFO 4486 --- [Collect all tenants major compactions5,3589673fc55cac40,3606428a5ba37f67] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:40:25.148 INFO 4486 --- [Collect all tenants major compactions5,3589673fc55cac40,3606428a5ba37f67] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:40:25.150 INFO 4486 --- [ocp-schedules-30,3589673fc55cac40,07462608b6db593f] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:40:25.153 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.153 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:40:25.169 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.170 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:40:25.170 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:40:25.172 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.177 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:40:25.177 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.178 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:40:25.186 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.186 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:40:25.190 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.191 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:40:25.194 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.194 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:25.202 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.203 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:40:25.203 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.203 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.203 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:25.204 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:40:25.207 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:40:25.211 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.212 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:40:25.214 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.215 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:25.220 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.221 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:40:25.222 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.223 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:40:25.224 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.224 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:25.234 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.235 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.235 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:40:25.236 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:40:25.239 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.240 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.240 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.241 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.242 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.242 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.243 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.244 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.244 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.244 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.245 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:40:25.245 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.246 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.246 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.247 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.247 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.248 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.248 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:40:25.249 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.250 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.250 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.251 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.251 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.252 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.252 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.253 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.253 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:40:25.253 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.254 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.255 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.255 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:40:25.258 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:40:25.266 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.267 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:40:25.270 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.272 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.275 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.276 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.277 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:25.278 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.278 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:40:25.278 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.280 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.280 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.282 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:40:25.282 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.284 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.286 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.286 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:40:25.287 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:25.288 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.290 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.291 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.292 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:40:25.292 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:40:25.293 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.294 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.296 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.297 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.300 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.300 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.301 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.301 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:25.304 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:40:25.306 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:40:25.312 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.313 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:25.338 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:25.339 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.340 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:40:25.345 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.346 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.349 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.350 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.351 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.351 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:40:25.351 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.351 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:40:25.351 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.351 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:40:25.353 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.353 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.355 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:40:25.355 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:40:25.356 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.358 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.361 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.362 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.364 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.365 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.367 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.368 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:40:25.371 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.373 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:40:25.375 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.376 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:40:25.378 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:40:25.381 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:40:25.443 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:25.444 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:25.445 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:40:25.457 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.457 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:40:25.457 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.457 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:40:25.457 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:40:25.457 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:40:25.461 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:40:25.461 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:40:25.994 INFO 4486 --- [prometheus-minute-schedule-1,c3d9b47b11c3c870,8c283b27028208ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:26.000 INFO 4486 --- [prometheus-minute-schedule-1,c3d9b47b11c3c870,8c283b27028208ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:27.124 INFO 4486 --- [ocp-schedules-28,cb3831ccf19a5dc4,76b8e449251933cf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:40:27.124+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:40:27.126 INFO 4486 --- [ocp-schedules-28,cb3831ccf19a5dc4,76b8e449251933cf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:40:27.127 INFO 4486 --- [ocp-schedules-28,cb3831ccf19a5dc4,76b8e449251933cf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:40:27.128 INFO 4486 --- [ocp-schedules-28,cb3831ccf19a5dc4,76b8e449251933cf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:40:27.140 INFO 4486 --- [ocp-async-6,18844faa69f54598,fb747f784e8657af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:27.400 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c7b1180952d0ecdc,52569f8cb934855a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c7b1180952d0ecdc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:27.400 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c7b1180952d0ecdc,52569f8cb934855a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c7b1180952d0ecdc, duration=1 ms] 2024-10-16 13:40:27.590 INFO 4486 --- [ocp-schedules-24,09e94ac15f354ed0,7296763134fb345e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:40:27.624 WARN 4486 --- [scanner-13267031-1,09e94ac15f354ed0,f5fc17a24d4bb3e4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:40:27.670 INFO 4486 --- [alarm-detect-6,09e94ac15f354ed0,03c4d95aaaa14715] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057200 ~ 1729057217 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057215, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057215, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057215=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:40:27.872 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:28.994 INFO 4486 --- [prometheus-minute-schedule-1,0d6052fd3b1cd417,a50e2bcc505c21d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:29.002 INFO 4486 --- [prometheus-minute-schedule-1,0d6052fd3b1cd417,a50e2bcc505c21d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:29.140 INFO 4486 --- [ocp-async-10,04006666d40e9e9c,f6c4e2d79ebc38a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:29.572 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4f111fa6129002cd,8d4f92998131ba72] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4f111fa6129002cd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:29.572 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4f111fa6129002cd,8d4f92998131ba72] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4f111fa6129002cd, duration=2 ms] 2024-10-16 13:40:30.103 INFO 4486 --- [prometheus-second-schedule-1,0e3188fc9ac93795,0dbecdd8b1f83e1f] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:40:30.543 INFO 4486 --- [ocp-schedules-20,2b11581650cf5d8b,c124b0f3093a4a1c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:40:30.873 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:31.134 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,44ee4115c831e554,a2402102a0fde1ec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=44ee4115c831e554, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:31.134 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,44ee4115c831e554,a2402102a0fde1ec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=44ee4115c831e554, duration=3 ms] 2024-10-16 13:40:31.141 INFO 4486 --- [ocp-async-3,ee3c5b75e3ac5667,047aca0ec77b67d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:31.454 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7bbee88fde201474,21974e9743a2ed95] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7bbee88fde201474, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:31.455 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7bbee88fde201474,21974e9743a2ed95] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7bbee88fde201474, duration=2 ms] 2024-10-16 13:40:31.994 INFO 4486 --- [prometheus-minute-schedule-1,87f41918b444ce2a,9eeb37abf293de1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:32.001 INFO 4486 --- [prometheus-minute-schedule-1,87f41918b444ce2a,9eeb37abf293de1b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:32.998 INFO 4486 --- [ocp-async-5,d291f449497dccfd,e855cd9f645dbb88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:40:33.141 INFO 4486 --- [ocp-async-7,78b3965a82328f49,5a87a661187b7834] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:33.874 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:34.994 INFO 4486 --- [prometheus-minute-schedule-1,fbfe160dd1d3f391,93afc49612f78179] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:35.001 INFO 4486 --- [prometheus-minute-schedule-1,fbfe160dd1d3f391,93afc49612f78179] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:35.141 INFO 4486 --- [ocp-async-2,33d7bda02d647bb0,105c6a59ec93689d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:36.875 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:37.063 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:40:37.064 INFO 4486 --- [ocp-schedules-29,420b495cd91daf11,03e3103a4fd9cc66] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:40:37.066 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:40:37.067 INFO 4486 --- [ocp-schedules-21,5fbf81ddfaca6e66,57eb104832f1f24a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:40:37.067 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:40:37.068 INFO 4486 --- [ocp-schedules-21,5fbf81ddfaca6e66,57eb104832f1f24a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:37.069 INFO 4486 --- [ocp-schedules-29,420b495cd91daf11,03e3103a4fd9cc66] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:40:37.081 INFO 4486 --- [ocp-schedules-12,fec0c9616ca4b3d5,4ac4c0f1a654542a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:40:37.082 INFO 4486 --- [ocp-schedules-12,fec0c9616ca4b3d5,4ac4c0f1a654542a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:37.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:40:37.090 INFO 4486 --- [ocp-schedules-5,949ae1a735135d1f,c6199c3099a76225] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:40:37.090 INFO 4486 --- [ocp-schedules-22,73aa0fd0b0dc0f51,cfd4045137c15e91] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:40:37.090 INFO 4486 --- [host-check-clockdiff1,73aa0fd0b0dc0f51,63678e6980519d3e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:40:37.090 INFO 4486 --- [host-check-clockdiff2,73aa0fd0b0dc0f51,41228c798141ac27] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:40:37.091 INFO 4486 --- [ocp-schedules-24,727a01722ae5ae3d,ba4303153c2277aa] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:40:37.091 INFO 4486 --- [host-check-clockdiff3,73aa0fd0b0dc0f51,df2a360c9cf41381] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:40:37.091 INFO 4486 --- [host-check-clockdiff4,73aa0fd0b0dc0f51,00d548f76f0c6af1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:40:37.091 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:40:37.091 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:40:37.091 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:40:37.092 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:40:37.093 INFO 4486 --- [host-check-agent-status2,727a01722ae5ae3d,75264b7e07559dae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:40:37.093 INFO 4486 --- [host-check-agent-status3,727a01722ae5ae3d,e931d19d11922131] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:40:37.093 INFO 4486 --- [host-check-agent-status1,727a01722ae5ae3d,40ff756a54e6f376] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:40:37.094 INFO 4486 --- [host-check-agent-status4,727a01722ae5ae3d,749004f671bb9260] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:40:37.106 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:37.106 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:40:37.128 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:40:37.134 INFO 4486 --- [host-check-agent-status2,727a01722ae5ae3d,75264b7e07559dae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:40:37.134 INFO 4486 --- [host-check-agent-status3,727a01722ae5ae3d,e931d19d11922131] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:40:37.136 INFO 4486 --- [host-check-agent-status4,727a01722ae5ae3d,749004f671bb9260] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:40:37.136 INFO 4486 --- [host-check-agent-status1,727a01722ae5ae3d,40ff756a54e6f376] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:37.136 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:37.136 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:40:37.136 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:40:37.143 INFO 4486 --- [ocp-async-5,8f6f78c3ca5e598c,bc58cd890583509d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:37.136 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:37.143 INFO 4486 --- [host-check-clockdiff3,73aa0fd0b0dc0f51,df2a360c9cf41381] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057237 3 3 , error= 2024-10-16 13:40:37.143 INFO 4486 --- [host-check-clockdiff1,73aa0fd0b0dc0f51,63678e6980519d3e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057237 0 0 , error= 2024-10-16 13:40:37.146 INFO 4486 --- [host-check-agent-status1,727a01722ae5ae3d,40ff756a54e6f376] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:37.146 INFO 4486 --- [host-check-agent-status3,727a01722ae5ae3d,e931d19d11922131] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:40:37.146 INFO 4486 --- [host-check-agent-status2,727a01722ae5ae3d,75264b7e07559dae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:40:37.147 INFO 4486 --- [host-check-agent-status4,727a01722ae5ae3d,749004f671bb9260] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:40:37.147 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:40:37.147 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:40:37.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:40:37.168 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:37.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.169 INFO 4486 --- [host-check-agent-status2,727a01722ae5ae3d,75264b7e07559dae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:40:37.170 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:37.171 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:40:37.171 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:40:37.173 INFO 4486 --- [host-check-agent-status3,727a01722ae5ae3d,e931d19d11922131] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:40:37.174 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:40:37.176447334+08:00 2024-10-16 13:40:37.174 INFO 4486 --- [host-check-agent-status1,727a01722ae5ae3d,40ff756a54e6f376] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:40:37.174 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:40:37.176106829+08:00 2024-10-16 13:40:37.174 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:40:37.177 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:40:37.177 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:40:37.174+08:00 2024-10-16 13:40:37.178 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:40:37.174+08:00 2024-10-16 13:40:37.178 INFO 4486 --- [host-check-agent-status3,727a01722ae5ae3d,e931d19d11922131] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:40:37.178 INFO 4486 --- [host-check-agent-status3,727a01722ae5ae3d,e931d19d11922131] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:40:37.178 INFO 4486 --- [host-check-agent-status3,727a01722ae5ae3d,e931d19d11922131] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:40:37.178 INFO 4486 --- [host-check-agent-status2,727a01722ae5ae3d,75264b7e07559dae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:40:37.178 INFO 4486 --- [host-check-agent-status2,727a01722ae5ae3d,75264b7e07559dae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:40:37.178 INFO 4486 --- [host-check-agent-status2,727a01722ae5ae3d,75264b7e07559dae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:40:37.178 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:40:37.179 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:40:37.179 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.180 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:40:37.182902597+08:00 2024-10-16 13:40:37.180 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:40:37.179825466+08:00 2024-10-16 13:40:37.180 INFO 4486 --- [host-check-agent-status4,727a01722ae5ae3d,749004f671bb9260] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:40:37.180 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:40:37.180 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:40:37.180+08:00 2024-10-16 13:40:37.180 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:40:37.180+08:00 2024-10-16 13:40:37.180 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:40:37.174+08:00 and status AVAILABLE 2024-10-16 13:40:37.181 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:40:37.174+08:00 and status AVAILABLE 2024-10-16 13:40:37.182 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.182 INFO 4486 --- [host-check-agent-status1,727a01722ae5ae3d,40ff756a54e6f376] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:40:37.182 INFO 4486 --- [host-check-agent-status1,727a01722ae5ae3d,40ff756a54e6f376] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:40:37.182 INFO 4486 --- [host-check-agent-status1,727a01722ae5ae3d,40ff756a54e6f376] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:40:37.183 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:40:37.183 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.183 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:40:37.180+08:00 and status AVAILABLE 2024-10-16 13:40:37.183 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:40:37.180+08:00 and status AVAILABLE 2024-10-16 13:40:37.184 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:40:37.184 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:40:37.184 INFO 4486 --- [host-check-agent-status4,727a01722ae5ae3d,749004f671bb9260] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:40:37.184 INFO 4486 --- [host-check-agent-status4,727a01722ae5ae3d,749004f671bb9260] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:40:37.184 INFO 4486 --- [host-check-agent-status4,727a01722ae5ae3d,749004f671bb9260] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:40:37.185 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:40:37.185 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:40:37.186 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.187 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:37.193 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:40:37.193 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:37.193 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:40:37.194 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:40:37.194 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:40:37.194 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:37.195 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:40:37.195 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:40:37.197 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.198 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:37.202 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.202 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.202 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.202 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.202 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.203 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.207 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.208 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:40:37.210 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.211 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:40:37.215 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:37.216 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:37.216 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:37.216 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:40:37.221 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.222 INFO 4486 --- [ocp-schedules-9,0abc8983cc53c24c,56ab83788ece01d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:40:37.223 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:40:37.247 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:40:37.250 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:40:37.255 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:40:37.258 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:40:37.265 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:40:37.266 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:40:37.266 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:40:37.275 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:37.276 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:40:37.280 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:37.280 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:37.292 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:40:37.307 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:40:37.313 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:40:37.316 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:40:37.323 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:40:37.327 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:40:37.331 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:37.336 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:37.337 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:40:37.349 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:40:37.351 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:37.358 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:40:37.359 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:40:37.369 INFO 4486 --- [host-check-host-status3,949ae1a735135d1f,421d7eecd9d888e4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:40:37.369 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:40:37.372 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:40:37.379 INFO 4486 --- [host-check-host-status4,949ae1a735135d1f,85d9031ca7a02ef0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:40:37.390 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:40:37.402 INFO 4486 --- [host-check-host-status2,949ae1a735135d1f,22ab53b05012d992] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:40:37.405 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8fc765871334b85f,c61b7a0e1514a7ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8fc765871334b85f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:37.405 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8fc765871334b85f,c61b7a0e1514a7ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8fc765871334b85f, duration=1 ms] 2024-10-16 13:40:37.427 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:37.429 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.430 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:40:37.431 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.432 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.432 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:40:37.433 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:40:37.444 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.444 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.444 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.444 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.444 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.444 INFO 4486 --- [host-check-host-status1,949ae1a735135d1f,656b6725400a57d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:40:37.444 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.473 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:40:37.512 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:40:37.557 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:37.572 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:37.574 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.575 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:40:37.576 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:40:37.577 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.577 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:40:37.578 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.579 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.580 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:40:37.589 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.589 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.589 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.589 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.589 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.590 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.597 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:40:37.724 INFO 4486 --- [ocp-schedules-27,20a8747e4404ef85,381424974f078fae] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:40:37.772 WARN 4486 --- [scanner-13267276-1,20a8747e4404ef85,b3c2c6f8cac11503] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:40:37.813 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:40:37.892 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:37.895 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.896 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:40:37.897 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.898 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.899 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:40:37.927 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.927 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.927 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.927 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.927 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:37.928 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:37.952 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:40:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:40:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:40:37.994 INFO 4486 --- [prometheus-minute-schedule-1,9f4989fc49394cee,3453b1cadefc346d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:37.999 INFO 4486 --- [prometheus-minute-schedule-1,9f4989fc49394cee,3453b1cadefc346d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:38.058 INFO 4486 --- [host-check-clockdiff4,73aa0fd0b0dc0f51,00d548f76f0c6af1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057238 3 3 , error= 2024-10-16 13:40:38.063 INFO 4486 --- [host-check-clockdiff2,73aa0fd0b0dc0f51,41228c798141ac27] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057238 3 3 , error= 2024-10-16 13:40:38.184 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:40:38.246 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:38.248 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:38.249 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:40:38.250 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:38.251 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:38.252 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:40:38.261 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:38.261 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:38.261 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:38.261 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:38.261 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:40:38.261 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:38.326 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:40:38.555 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:40:38.329872, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:40:38.556 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:40:38.329872, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:40:38.604 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:40:38.329872} } 2024-10-16 13:40:38.661 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:40:38.666 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:40:38.670 INFO 4486 --- [ocp-schedules-15,f17e8f28482a7cb5,253639ffa7f23486] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:40:38.956 INFO 4486 --- [ocp-schedules-23,cd7c5db070ddcb5c,cf4fa0a538223771] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:40:39.143 INFO 4486 --- [ocp-async-7,5e23e6d74e19b081,6fd601f86fe685d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:39.579 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,94e2c69438d839a6,28a6908b14076591] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=94e2c69438d839a6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:39.580 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,94e2c69438d839a6,28a6908b14076591] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=94e2c69438d839a6, duration=1 ms] 2024-10-16 13:40:39.876 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:40.006 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:40:40.006 INFO 4486 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:40:40.029 INFO 4486 --- [Check obproxy1,29a631378fab631a,d1a1a83d982ab3ec] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:40:40.029 INFO 4486 --- [Check obproxy1,29a631378fab631a,d1a1a83d982ab3ec] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:40.031 INFO 4486 --- [Collect obproxy1,e4c7ecf8dc7f595e,2a1120764890de25] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:40:40.031 INFO 4486 --- [Collect obproxy1,e4c7ecf8dc7f595e,2a1120764890de25] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:40.032 INFO 4486 --- [Check obproxy2,c6fe79b1c5014173,ab8e528de9225d46] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:40:40.032 INFO 4486 --- [Check obproxy2,c6fe79b1c5014173,ab8e528de9225d46] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:40.035 INFO 4486 --- [Collect obproxy2,901c239cb003d82c,44e76fa0e0a4963e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:40:40.036 INFO 4486 --- [Collect obproxy2,901c239cb003d82c,44e76fa0e0a4963e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:40:40.050 INFO 4486 --- [Check obproxy1,29a631378fab631a,d1a1a83d982ab3ec] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:40:40.050 INFO 4486 --- [Check obproxy1,29a631378fab631a,d1a1a83d982ab3ec] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:40:40.054 INFO 4486 --- [Check obproxy2,c6fe79b1c5014173,ab8e528de9225d46] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:40:40.054 INFO 4486 --- [Check obproxy2,c6fe79b1c5014173,ab8e528de9225d46] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:40:40.055 INFO 4486 --- [Collect obproxy2,901c239cb003d82c,44e76fa0e0a4963e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:40:40.059 INFO 4486 --- [Collect obproxy1,e4c7ecf8dc7f595e,2a1120764890de25] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:40:40.061 INFO 4486 --- [Collect obproxy2,901c239cb003d82c,44e76fa0e0a4963e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:40:40.064 INFO 4486 --- [Collect obproxy1,e4c7ecf8dc7f595e,2a1120764890de25] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:40:40.066 INFO 4486 --- [Collect obproxy2,901c239cb003d82c,44e76fa0e0a4963e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:40:40.070 INFO 4486 --- [Collect obproxy1,e4c7ecf8dc7f595e,2a1120764890de25] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:40:40.418 INFO 4486 --- [ocp-schedules-16,d178e31092876514,424de0a04f0a051a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:40:40.418 INFO 4486 --- [ocp-schedules-16,4d624122caa24852,870ce93ea35f3f6a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:40:40.420 INFO 4486 --- [ocp-schedules-16,4d624122caa24852,870ce93ea35f3f6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:40:40.455 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.456 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:40.469 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.469 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:40.495 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:40.496 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.496 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:40:40.504 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.504 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:40.525 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:40.526 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.526 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:40.527 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:40:40.528 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.530 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:40:40.532 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:40.541 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.541 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:40.551 INFO 4486 --- [ocp-schedules-14,ac81f7200e964da2,3800f770b3c3cb29] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:40:40.555 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.567 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:40.569 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.570 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:40.571 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:40:40.576 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:40:40.580 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:40.582 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:40.609 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:40:40.635 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:40.637 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.638 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:40:40.649 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:40:40.649 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:40:40.684 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:40:40.692 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:40.694 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.695 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:40:40.695 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:40.696 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.696 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:40:40.699 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.702 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:40:40.706 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:40.706 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:40:40.706 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:40:40.736 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.736 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:40.805 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.805 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:40.829 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.829 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:40.856 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.856 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:40:40.882 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.882 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:40:40.912 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:40:40.940 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:40.942 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:40.942 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:40:40.947 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:40:40.952 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:40:40.956 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:40:40.958 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:40:40.979 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:40:40.994 INFO 4486 --- [prometheus-minute-schedule-1,8bbed428af84ed0f,8d2636aa012f40bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:40.995 INFO 4486 --- [prometheus-hour-schedule-1,71ba08c34bdb68bd,1f6a1d1c5e4df740] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:40:41.000 INFO 4486 --- [prometheus-minute-schedule-1,8bbed428af84ed0f,8d2636aa012f40bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:41.121 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:40:41.140 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,53577bf737dc863a,4901829510fcdf99] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=53577bf737dc863a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:41.140 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,53577bf737dc863a,4901829510fcdf99] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=53577bf737dc863a, duration=1 ms] 2024-10-16 13:40:41.143 INFO 4486 --- [ocp-async-2,b6f11fef166995e8,37746839f1be04f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:41.150 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:40:41.151 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:40:41.152 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:40:41.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:40:41.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:40:41.460 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,67a857a576317b53,c2da67edb5e8b3fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=67a857a576317b53, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:41.460 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,67a857a576317b53,c2da67edb5e8b3fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=67a857a576317b53, duration=1 ms] 2024-10-16 13:40:42.105 INFO 4486 --- [ocp-schedules-5,0ae3fbb76a995182,5e3227fb19d509f4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:40:42.877 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:42.998 INFO 4486 --- [ocp-async-2,bf5d500f09cc01c4,1b77e67b0d57fc13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:40:43.144 INFO 4486 --- [ocp-async-9,3c0b1c6faac3fdd6,458823e355c1fa07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:43.994 INFO 4486 --- [prometheus-minute-schedule-1,27b28b47598cb93f,ccc7e69980a6b69f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:44.005 INFO 4486 --- [prometheus-minute-schedule-1,27b28b47598cb93f,ccc7e69980a6b69f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:45.144 INFO 4486 --- [ocp-async-8,3bfecebea893a982,2640592c2a3c6bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:45.480 INFO 4486 --- [ocp-schedules-7,6a1941b2b25022e5,346083a810d73431] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:40:45.486 INFO 4486 --- [ocp-schedules-7,6a1941b2b25022e5,346083a810d73431] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:40:45.486 INFO 4486 --- [ocp-schedules-7,6a1941b2b25022e5,346083a810d73431] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:40:45.501 INFO 4486 --- [ocp-schedules-3,3ca49b830abe5f6b,465348c724876c6e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:40:45.501 INFO 4486 --- [ocp-schedules-3,3ca49b830abe5f6b,465348c724876c6e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:40:45.878 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:46.994 INFO 4486 --- [prometheus-minute-schedule-1,fbb0c79a58b6ecf0,84bfc0d72109b728] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:47.001 INFO 4486 --- [prometheus-minute-schedule-1,fbb0c79a58b6ecf0,84bfc0d72109b728] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:47.070 INFO 4486 --- [ocp-schedules-21,5fbf81ddfaca6e66,57eb104832f1f24a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:40:47.133 INFO 4486 --- [ocp-schedules-28,65b296c3c8e6a6f7,cfa3e5b95b9263db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:40:47.145 INFO 4486 --- [ocp-async-4,4fcfa3c4adea084a,86b03de651505ec2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:47.178 INFO 4486 --- [ocp-schedules-12,fec0c9616ca4b3d5,4ac4c0f1a654542a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:40:47.412 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ec5345488c83b262,2809c4f005ae5240] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ec5345488c83b262, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:47.412 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ec5345488c83b262,2809c4f005ae5240] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ec5345488c83b262, duration=3 ms] 2024-10-16 13:40:47.942 INFO 4486 --- [ocp-schedules-13,91d7a8490f5dcc63,6b0d52fc9dce989b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:40:47.967 WARN 4486 --- [scanner-13267606-1,91d7a8490f5dcc63,68046b25555fa2b5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:40:48.037 INFO 4486 --- [alarm-detect-19,91d7a8490f5dcc63,dfd96a1b1fb859ea] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057215 ~ 1729057238 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057230, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057230, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057230=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:40:48.879 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:49.145 INFO 4486 --- [ocp-async-9,43a1de24f5e2068a,e3737b1af46d2f8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:49.584 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ff8509e0789d71de,a2cd90b900383020] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ff8509e0789d71de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:49.584 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ff8509e0789d71de,a2cd90b900383020] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ff8509e0789d71de, duration=1 ms] 2024-10-16 13:40:49.994 INFO 4486 --- [prometheus-minute-schedule-1,aefed079d46f926e,dec1320b2fc51954] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:49.999 INFO 4486 --- [prometheus-minute-schedule-1,aefed079d46f926e,dec1320b2fc51954] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:50.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cafa6fa94063b96f,afb788b50259dd66] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=cafa6fa94063b96f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:40:50.187 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cafa6fa94063b96f,afb788b50259dd66] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=cafa6fa94063b96f, duration=2 ms] 2024-10-16 13:40:50.350 INFO 4486 --- [ocp-schedules-3,80e89c589c5f3940,b0302f800c824d3d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:40:50.350 INFO 4486 --- [ocp-schedules-3,80e89c589c5f3940,b0302f800c824d3d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:40:50.350 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:40:50.351 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:40:50.351 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:40:50.351 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:40:50.351 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:40:50.357 INFO 4486 --- [ocp-schedules-3,80e89c589c5f3940,b0302f800c824d3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:40:50.357 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:40:50.357 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:40:50.357 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:40:50.357 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:40:50.358 INFO 4486 --- [ocp-schedules-3,80e89c589c5f3940,b0302f800c824d3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:40:50.358 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:40:50.359 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:40:50.359 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:40:50.359 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:40:50.361 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:40:50.363 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:40:50.384 INFO 4486 --- [ocp-schedules-3,80e89c589c5f3940,b0302f800c824d3d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:40:50.384 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:40:50.384 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:40:50.389 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:40:50.389 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:40:50.407 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:40:50.412 INFO 4486 --- [ocp-schedules-3,80e89c589c5f3940,b0302f800c824d3d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:40:50.418 INFO 4486 --- [ocp-schedules-7,a0fd405911f97347,90e3dddcd1ec7f2e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:40:50.418 INFO 4486 --- [ocp-schedules-7,a0fd405911f97347,90e3dddcd1ec7f2e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:40:50.420 INFO 4486 --- [ocp-schedules-7,a0fd405911f97347,90e3dddcd1ec7f2e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:40:50.420 INFO 4486 --- [ocp-schedules-7,a0fd405911f97347,90e3dddcd1ec7f2e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:40:50.421 INFO 4486 --- [ocp-schedules-7,a0fd405911f97347,90e3dddcd1ec7f2e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:40:50.421 INFO 4486 --- [ocp-schedules-7,a0fd405911f97347,90e3dddcd1ec7f2e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:40:50.556 INFO 4486 --- [ocp-schedules-30,657cc1e8f914106c,5829655c081afcd8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:40:51.145 INFO 4486 --- [ocp-async-7,5220e214e22bfd06,820935936825b9a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:51.146 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a4c07131c29d02cb,e8f6e43d1fc6f983] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a4c07131c29d02cb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:51.146 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a4c07131c29d02cb,e8f6e43d1fc6f983] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a4c07131c29d02cb, duration=1 ms] 2024-10-16 13:40:51.464 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d675e164205a55cc,9ec8f32dd41be432] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d675e164205a55cc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:51.464 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d675e164205a55cc,9ec8f32dd41be432] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d675e164205a55cc, duration=1 ms] 2024-10-16 13:40:51.880 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:52.994 INFO 4486 --- [prometheus-minute-schedule-1,9d54a76da8962688,69d46c5f35883e03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:52.998 INFO 4486 --- [ocp-async-7,51cea90e3948ebfb,b79184b6129eb395] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:40:53.000 INFO 4486 --- [prometheus-minute-schedule-1,9d54a76da8962688,69d46c5f35883e03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:53.145 INFO 4486 --- [ocp-async-1,11fdffd36a8aa096,49d48f2b72597e98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:54.881 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:55.146 INFO 4486 --- [ocp-async-3,ed2dfcfb362427c1,fa5e59406ad7d028] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:55.994 INFO 4486 --- [prometheus-minute-schedule-1,fad9acee6a42a230,1e8eccc3117f5816] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:56.001 INFO 4486 --- [prometheus-minute-schedule-1,fad9acee6a42a230,1e8eccc3117f5816] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:57.146 INFO 4486 --- [ocp-async-8,df4549e618281a29,dbebaa089a060e66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:57.420 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,344fbc71d54d52e7,ef1f7e5c0961d9ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=344fbc71d54d52e7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:57.420 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,344fbc71d54d52e7,ef1f7e5c0961d9ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=344fbc71d54d52e7, duration=1 ms] 2024-10-16 13:40:57.882 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:40:58.092 INFO 4486 --- [ocp-schedules-20,d77f51fe55a05908,f6a5000d8fc6ce19] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:40:58.117 WARN 4486 --- [scanner-13267824-1,d77f51fe55a05908,37fef0c14eba8842] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:40:58.229 INFO 4486 --- [alarm-detect-14,d77f51fe55a05908,b745d98d390179a9] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057230 ~ 1729057248 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057245, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057245, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057245=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:40:58.960 INFO 4486 --- [ocp-schedules-15,ed1ea68b1be65033,c4553135eec96e59] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:40:58.994 INFO 4486 --- [prometheus-minute-schedule-1,a4aade51c76f7a7d,3fefc42c04be1a14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057260, interval=60 2024-10-16 13:40:59.001 INFO 4486 --- [prometheus-minute-schedule-1,a4aade51c76f7a7d,3fefc42c04be1a14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:40:59.146 INFO 4486 --- [ocp-async-4,fb60a3b949d5d590,25a0e0bd7f338005] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:40:59.589 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,115951e2c8499d84,6d67d7d994217851] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=115951e2c8499d84, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:40:59.589 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,115951e2c8499d84,6d67d7d994217851] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=115951e2c8499d84, duration=1 ms] 2024-10-16 13:41:00.561 INFO 4486 --- [ocp-schedules-23,008e0167952134b3,7723cb4dca046534] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:41:00.883 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:01.146 INFO 4486 --- [ocp-async-1,23b8e93bac3f4a6d,0f6351108d22a7b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:01.153 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ab05f43730d5f671,1f6b003000dfd824] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ab05f43730d5f671, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:01.153 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ab05f43730d5f671,1f6b003000dfd824] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ab05f43730d5f671, duration=2 ms] 2024-10-16 13:41:01.468 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f822a77df735c442,6dce0967802965bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f822a77df735c442, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:01.468 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f822a77df735c442,6dce0967802965bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f822a77df735c442, duration=1 ms] 2024-10-16 13:41:01.515 INFO 4486 --- [ocp-schedules-24,2cdcaa897dadd164,64d5eff09c51cd3e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:41:01.515+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:41:01.520 INFO 4486 --- [ocp-schedules-24,2cdcaa897dadd164,64d5eff09c51cd3e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:41:01.525 INFO 4486 --- [ocp-schedules-24,2cdcaa897dadd164,64d5eff09c51cd3e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:41:01.536 INFO 4486 --- [ocp-schedules-24,2cdcaa897dadd164,64d5eff09c51cd3e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:41:01.994 INFO 4486 --- [prometheus-minute-schedule-1,a517e234da77df66,b4031c98e3171482] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:02.000 INFO 4486 --- [prometheus-minute-schedule-1,a517e234da77df66,b4031c98e3171482] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:02.999 INFO 4486 --- [ocp-async-9,7e11313aac92ee17,667cd1c8f8667da3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:41:03.100 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,11814da229ee264f,e6d3146ab488eb65] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=11814da229ee264f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:41:03.100 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,11814da229ee264f,e6d3146ab488eb65] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=11814da229ee264f, duration=2 ms] 2024-10-16 13:41:03.147 INFO 4486 --- [ocp-async-6,8011a22463ed0334,0c81cc087443c638] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:03.248 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,8fa7cbbeef891d38,0430829ebd2d0429] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=8fa7cbbeef891d38, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:41:03.248 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,8fa7cbbeef891d38,0430829ebd2d0429] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=8fa7cbbeef891d38, duration=1 ms] 2024-10-16 13:41:03.884 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:04.039 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:41:04.039 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:41:04.046 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:41:04.046 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:41:04.046 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:41:04.046 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:41:04.053 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:41:04.053 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:41:04.053 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:41:04.053 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:41:04.095 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:41:04.102 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:41:04.102 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:41:04.102 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:41:04.102 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:41:04.102 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:41:04.108 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:41:04.514 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:41:04.994 INFO 4486 --- [prometheus-minute-schedule-1,06aae3ec5fa8f868,8bf0aa7d354c14bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:05.001 INFO 4486 --- [prometheus-minute-schedule-1,06aae3ec5fa8f868,8bf0aa7d354c14bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:05.097 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.044626874S 2024-10-16 13:41:05.098 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:41:05.147 INFO 4486 --- [ocp-async-10,b5d60db5fafed1e7,3304e97f6efd0003] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:05.175 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:41:05.175 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:41:05.202 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:41:05.276 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:41:05.276 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:41:05.276 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:41:05.276 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:41:05.276 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:41:05.276 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:41:05.287 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:41:05.417 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] o.s.boot.SpringApplication : Started application in 1.371 seconds (JVM running for 78168.468) 2024-10-16 13:41:05.417 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.371433357S 2024-10-16 13:41:05.789 INFO 4486 --- [ocp-schedules-20,7644589b933024a3,a811bd563e8d66d3] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:41:06.885 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:07.066 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:41:07.067 INFO 4486 --- [ocp-schedules-27,81f3aa258ef19029,67bf3a5cdd3b9c22] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:41:07.069 INFO 4486 --- [ocp-schedules-27,81f3aa258ef19029,67bf3a5cdd3b9c22] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:07.081 INFO 4486 --- [ocp-schedules-24,37d2b0c5bc378b87,f9cbd1209f10effb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:41:07.084 INFO 4486 --- [ocp-schedules-24,37d2b0c5bc378b87,f9cbd1209f10effb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:07.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:41:07.089 INFO 4486 --- [ocp-schedules-6,fc0e301650dcbf5d,0dd8fa9b95447c6c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:41:07.089 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:41:07.090 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:41:07.090 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:41:07.091 INFO 4486 --- [ocp-schedules-30,7da0430ca7b3c408,df95db84d68722b7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:41:07.090 INFO 4486 --- [ocp-schedules-1,f9594cd77a6759bb,498774513d13c4f7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:41:07.091 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:41:07.091 INFO 4486 --- [host-check-agent-status1,7da0430ca7b3c408,48b84dae2ddbb51b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:41:07.092 INFO 4486 --- [host-check-agent-status3,7da0430ca7b3c408,872b9b980f071d58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:41:07.092 INFO 4486 --- [host-check-agent-status2,7da0430ca7b3c408,633ef9ce20de8483] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:41:07.094 INFO 4486 --- [host-check-clockdiff1,f9594cd77a6759bb,b6165f940b28df93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:41:07.094 INFO 4486 --- [host-check-clockdiff2,f9594cd77a6759bb,e86c002b6b5e6897] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:41:07.094 INFO 4486 --- [host-check-clockdiff3,f9594cd77a6759bb,e1b46d061ef55c84] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:41:07.094 INFO 4486 --- [host-check-agent-status4,7da0430ca7b3c408,592f38cbf1124b04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:41:07.096 INFO 4486 --- [host-check-clockdiff4,f9594cd77a6759bb,2245d4c2687ff61d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:41:07.102 INFO 4486 --- [host-check-agent-status3,7da0430ca7b3c408,872b9b980f071d58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:41:07.102 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:41:07.102 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:41:07.102 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:07.103 INFO 4486 --- [host-check-agent-status1,7da0430ca7b3c408,48b84dae2ddbb51b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:07.103 INFO 4486 --- [host-check-agent-status4,7da0430ca7b3c408,592f38cbf1124b04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:41:07.109 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:41:07.113 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:07.122 INFO 4486 --- [host-check-agent-status1,7da0430ca7b3c408,48b84dae2ddbb51b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:07.122 INFO 4486 --- [host-check-agent-status4,7da0430ca7b3c408,592f38cbf1124b04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:41:07.122 INFO 4486 --- [host-check-agent-status3,7da0430ca7b3c408,872b9b980f071d58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:41:07.129 INFO 4486 --- [host-check-agent-status2,7da0430ca7b3c408,633ef9ce20de8483] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:41:07.129 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:41:07.129 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:41:07.129 INFO 4486 --- [host-check-clockdiff4,f9594cd77a6759bb,2245d4c2687ff61d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057267 3 3 , error= 2024-10-16 13:41:07.130 INFO 4486 --- [host-check-clockdiff2,f9594cd77a6759bb,e86c002b6b5e6897] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057267 3 3 , error= 2024-10-16 13:41:07.132 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:41:07.133 INFO 4486 --- [host-check-clockdiff3,f9594cd77a6759bb,e1b46d061ef55c84] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057267 3 3 , error= 2024-10-16 13:41:07.133 INFO 4486 --- [host-check-clockdiff1,f9594cd77a6759bb,b6165f940b28df93] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057267 0 0 , error= 2024-10-16 13:41:07.135 INFO 4486 --- [host-check-agent-status2,7da0430ca7b3c408,633ef9ce20de8483] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:41:07.144 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.145 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:07.146 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:41:07.147 INFO 4486 --- [ocp-async-5,6096758012dcb916,d0ee5704b33db23b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:07.160 INFO 4486 --- [host-check-agent-status3,7da0430ca7b3c408,872b9b980f071d58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:41:07.161 INFO 4486 --- [host-check-agent-status1,7da0430ca7b3c408,48b84dae2ddbb51b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:41:07.162 INFO 4486 --- [host-check-agent-status4,7da0430ca7b3c408,592f38cbf1124b04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:41:07.162 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:41:07.162 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:41:07.162 INFO 4486 --- [host-check-agent-status2,7da0430ca7b3c408,633ef9ce20de8483] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:41:07.166 INFO 4486 --- [host-check-agent-status3,7da0430ca7b3c408,872b9b980f071d58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:41:07.166 INFO 4486 --- [host-check-agent-status3,7da0430ca7b3c408,872b9b980f071d58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:41:07.166 INFO 4486 --- [host-check-agent-status3,7da0430ca7b3c408,872b9b980f071d58] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:41:07.166 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:41:07.169249317+08:00 2024-10-16 13:41:07.167 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:41:07.170344241+08:00 2024-10-16 13:41:07.167 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:41:07.166+08:00 2024-10-16 13:41:07.167 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.168 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:41:07.167+08:00 2024-10-16 13:41:07.168 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:41:07.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status2,7da0430ca7b3c408,633ef9ce20de8483] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status4,7da0430ca7b3c408,592f38cbf1124b04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status2,7da0430ca7b3c408,633ef9ce20de8483] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status4,7da0430ca7b3c408,592f38cbf1124b04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status1,7da0430ca7b3c408,48b84dae2ddbb51b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status4,7da0430ca7b3c408,592f38cbf1124b04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status2,7da0430ca7b3c408,633ef9ce20de8483] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status1,7da0430ca7b3c408,48b84dae2ddbb51b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:41:07.171 INFO 4486 --- [host-check-agent-status1,7da0430ca7b3c408,48b84dae2ddbb51b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:41:07.172 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:41:07.173 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:41:07.176155433+08:00 2024-10-16 13:41:07.173 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:41:07.172751948+08:00 2024-10-16 13:41:07.173 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:41:07.173+08:00 2024-10-16 13:41:07.174 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:41:07.173+08:00 2024-10-16 13:41:07.175 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:41:07.178 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:41:07.166+08:00 and status AVAILABLE 2024-10-16 13:41:07.179 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:41:07.167+08:00 and status AVAILABLE 2024-10-16 13:41:07.180 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:41:07.182 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:41:07.183 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:41:07.173+08:00 and status AVAILABLE 2024-10-16 13:41:07.185 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:41:07.173+08:00 and status AVAILABLE 2024-10-16 13:41:07.185 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:41:07.186 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.187 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:41:07.190 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:41:07.190 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:41:07.191 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:41:07.192 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:41:07.192 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:41:07.194 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:41:07.195 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:41:07.196 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:07.198 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:07.203 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.204 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:41:07.214 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:07.214 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:07.215 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.216 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:41:07.217 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:07.224 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:07.229 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.230 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:41:07.233 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.234 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:41:07.243 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:41:07.244 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:07.246 INFO 4486 --- [ocp-schedules-22,6aa1547525c715ee,f41d0abfc8237686] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:41:07.250 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:41:07.255 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:41:07.262 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:41:07.263 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:41:07.274 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:41:07.282 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:07.282 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:41:07.283 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:07.291 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:07.292 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:41:07.307 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:07.311 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:41:07.317 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:41:07.323 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:41:07.328 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:41:07.338 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:07.340 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:41:07.345 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:07.350 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:41:07.360 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:41:07.367 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:41:07.370 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:07.373 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:41:07.377 INFO 4486 --- [host-check-host-status3,fc0e301650dcbf5d,c3d50839c8244b16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:41:07.378 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:41:07.390 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:07.394 INFO 4486 --- [host-check-host-status4,fc0e301650dcbf5d,bdda57bc8a6a5320] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:41:07.406 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:41:07.418 INFO 4486 --- [host-check-host-status2,fc0e301650dcbf5d,ae7dc6d3cf4a56bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:41:07.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e74b36106ea3bd31,ea10ceee3e0136c7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e74b36106ea3bd31, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:07.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e74b36106ea3bd31,ea10ceee3e0136c7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e74b36106ea3bd31, duration=2 ms] 2024-10-16 13:41:07.443 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:41:07.454 INFO 4486 --- [host-check-host-status1,fc0e301650dcbf5d,0969e179b6a43ad7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:41:07.994 INFO 4486 --- [prometheus-minute-schedule-1,ae4bd8f1f9055927,1e1a8af671385d9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:07.999 INFO 4486 --- [prometheus-minute-schedule-1,ae4bd8f1f9055927,1e1a8af671385d9c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:08.338 INFO 4486 --- [ocp-schedules-11,b9dc9a3758de5cd2,b172a7eebd04ba6b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:41:08.364 WARN 4486 --- [scanner-13268142-1,b9dc9a3758de5cd2,861af9580c6a4602] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:41:09.147 INFO 4486 --- [ocp-async-3,3105d785aec9ba6a,62c510d7d12607be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:09.594 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6f2549c2088f6735,82b5fe0f8906f08b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6f2549c2088f6735, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:09.595 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6f2549c2088f6735,82b5fe0f8906f08b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6f2549c2088f6735, duration=1 ms] 2024-10-16 13:41:09.886 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:10.006 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:41:10.006 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:41:10.024 INFO 4486 --- [Check obproxy1,d88198df7cb3ce56,03760960fa401b4b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:41:10.024 INFO 4486 --- [Check obproxy2,0412fc9bcb27678b,f97f3a7345af1808] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:41:10.028 INFO 4486 --- [Check obproxy1,d88198df7cb3ce56,03760960fa401b4b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:10.029 INFO 4486 --- [Collect obproxy2,b4d93444788ad060,e9914c5ad94c3ae3] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:41:10.029 INFO 4486 --- [Collect obproxy2,b4d93444788ad060,e9914c5ad94c3ae3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:10.030 INFO 4486 --- [Collect obproxy1,39b612e84f99ef91,ab5bef0c6c0ce15a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:41:10.030 INFO 4486 --- [Collect obproxy1,39b612e84f99ef91,ab5bef0c6c0ce15a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:10.030 INFO 4486 --- [Check obproxy2,0412fc9bcb27678b,f97f3a7345af1808] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:10.051 INFO 4486 --- [Check obproxy2,0412fc9bcb27678b,f97f3a7345af1808] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:41:10.052 INFO 4486 --- [Check obproxy2,0412fc9bcb27678b,f97f3a7345af1808] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:41:10.054 INFO 4486 --- [Collect obproxy2,b4d93444788ad060,e9914c5ad94c3ae3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:41:10.059 INFO 4486 --- [Check obproxy1,d88198df7cb3ce56,03760960fa401b4b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:41:10.059 INFO 4486 --- [Check obproxy1,d88198df7cb3ce56,03760960fa401b4b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:41:10.059 INFO 4486 --- [Collect obproxy2,b4d93444788ad060,e9914c5ad94c3ae3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:41:10.063 INFO 4486 --- [Collect obproxy1,39b612e84f99ef91,ab5bef0c6c0ce15a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:41:10.063 INFO 4486 --- [Collect obproxy2,b4d93444788ad060,e9914c5ad94c3ae3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:41:10.066 INFO 4486 --- [Collect obproxy1,39b612e84f99ef91,ab5bef0c6c0ce15a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:41:10.068 INFO 4486 --- [Collect obproxy1,39b612e84f99ef91,ab5bef0c6c0ce15a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:41:10.566 INFO 4486 --- [ocp-schedules-2,71270a76ce783aea,9de83a518e65fd7e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:41:10.979 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:41:10.994 INFO 4486 --- [prometheus-minute-schedule-1,135329bae2df43a9,6153f2bc6f35df1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:10.999 INFO 4486 --- [prometheus-minute-schedule-1,135329bae2df43a9,6153f2bc6f35df1e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:41:11.148 INFO 4486 --- [ocp-async-4,7334eed11fb96699,afea79e52e6f7408] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:11.158 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e0b375be6bdb139e,9443cf3bbfdfd22d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e0b375be6bdb139e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:11.158 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e0b375be6bdb139e,9443cf3bbfdfd22d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e0b375be6bdb139e, duration=1 ms] 2024-10-16 13:41:11.181 INFO 4486 --- [ocp-schedules-16,4d624122caa24852,870ce93ea35f3f6a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:41:11.181 INFO 4486 --- [ocp-schedules-16,7efd3d024ab34ea9,dff3770f84abffcf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:41:11.183 INFO 4486 --- [ocp-schedules-16,7efd3d024ab34ea9,dff3770f84abffcf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:11.216 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.217 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:11.220 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.221 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:11.253 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.253 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.254 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:11.257 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.258 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:41:11.286 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.287 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.287 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:11.289 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:41:11.293 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.296 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:41:11.297 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:11.310 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.310 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:11.326 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.334 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.335 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.335 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:11.336 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:41:11.341 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:41:11.345 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:11.347 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:11.364 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.366 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.367 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:41:11.369 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:41:11.378 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:41:11.378 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:41:11.416 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.418 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.419 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:11.422 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.425 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:41:11.429 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:11.442 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:41:11.451 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.455 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.455 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:11.456 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.457 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:41:11.466 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:41:11.466 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:41:11.479 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0b567ba2ac034781,9aa8cc8be93b62cd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0b567ba2ac034781, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:11.479 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0b567ba2ac034781,9aa8cc8be93b62cd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0b567ba2ac034781, duration=1 ms] 2024-10-16 13:41:11.482 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.482 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:11.506 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.506 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:11.531 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.531 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:11.554 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.554 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:11.580 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:11.604 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.605 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.606 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:11.610 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:41:11.615 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:41:11.620 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:41:11.621 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:41:11.641 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:41:11.788 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:41:11.824 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:11.826 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:11.827 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:41:11.858 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:41:11.858 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:41:12.888 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:12.999 INFO 4486 --- [ocp-async-10,30acebeae702171e,ed25fad4ccd64841] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:41:13.148 INFO 4486 --- [ocp-async-5,723e8ef5efe658af,a1932117f341bc32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:13.558 ERROR 4486 --- [metric-archive-second-16,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2323956) Duplicate entry '669741696473960448-1729057200' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:41:13.556635] [YB420AC531B8-00060B7D84EAAF75-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '669741696473960448-1729057200' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:41:13.556635] [YB420AC531B8-00060B7D84EAAF75-0-0] 2024-10-16 13:41:13.578 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057200, nodeCount=1000, affectRows=1000 2024-10-16 13:41:13.582 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057200, nodeCount=1000, validCount=1000 2024-10-16 13:41:13.586 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057200, nodeCount=1000, affectRows=1000 2024-10-16 13:41:13.589 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057200, nodeCount=999, validCount=999 2024-10-16 13:41:13.597 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057200, nodeCount=1000, affectRows=1000 2024-10-16 13:41:13.602 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057200, nodeCount=1000, validCount=1000 2024-10-16 13:41:13.789 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057200, nodeCount=449, affectRows=449 2024-10-16 13:41:13.791 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057200, nodeCount=447, validCount=447 2024-10-16 13:41:13.994 INFO 4486 --- [prometheus-minute-schedule-1,01b59409dc20f629,29b5739fc4a3bc62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:14.000 INFO 4486 --- [prometheus-minute-schedule-1,01b59409dc20f629,29b5739fc4a3bc62] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:14.678 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057200, nodeCount=480, affectRows=480 2024-10-16 13:41:15.148 INFO 4486 --- [ocp-async-6,321579f4de3cf8d0,cabc3f67906da0ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:15.387 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057200, nodeCount=480, validCount=480 2024-10-16 13:41:15.889 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:16.994 INFO 4486 --- [prometheus-minute-schedule-1,9142103de3d4cf86,e7801001ca6d6410] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:17.000 INFO 4486 --- [prometheus-minute-schedule-1,9142103de3d4cf86,e7801001ca6d6410] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:17.070 INFO 4486 --- [ocp-schedules-27,81f3aa258ef19029,67bf3a5cdd3b9c22] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:41:17.137 INFO 4486 --- [ocp-schedules-23,cb5eca1df66adec8,328ee6caa2bd93f6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:41:17.148 INFO 4486 --- [ocp-async-3,a4bd3a7f6f3a0ce2,9386c6fc841f5e07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:17.176 INFO 4486 --- [ocp-schedules-24,37d2b0c5bc378b87,f9cbd1209f10effb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:41:17.432 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,65b83e1eab697aa3,b6456aaf9ed70d7d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=65b83e1eab697aa3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:17.432 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,65b83e1eab697aa3,b6456aaf9ed70d7d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=65b83e1eab697aa3, duration=2 ms] 2024-10-16 13:41:18.472 INFO 4486 --- [ocp-schedules-28,1abdd02abbd7f12c,a6e242a3a967b795] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:41:18.492 WARN 4486 --- [scanner-13269203-1,1abdd02abbd7f12c,9d634fa08e60da05] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:41:18.590 INFO 4486 --- [alarm-detect-14,1abdd02abbd7f12c,afdd734acfccd53d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057245 ~ 1729057268 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057260, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057260, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057260=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:41:18.597 INFO 4486 --- [alarm-detect-14,1abdd02abbd7f12c,afdd734acfccd53d] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:41:18.891 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:18.964 INFO 4486 --- [ocp-schedules-14,b25c943c812d0a1f,ff59e1174f117bdf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:41:19.149 INFO 4486 --- [ocp-async-2,623c570390e930e5,533f8a5cc5f0a03f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:19.600 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ea724d7febd5d5e6,c6249a6c56d73380] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ea724d7febd5d5e6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:19.600 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ea724d7febd5d5e6,c6249a6c56d73380] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ea724d7febd5d5e6, duration=1 ms] 2024-10-16 13:41:19.994 INFO 4486 --- [prometheus-minute-schedule-1,46ec13dc0bd66c6a,5f325cf4897f8c8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:20.002 INFO 4486 --- [prometheus-minute-schedule-1,46ec13dc0bd66c6a,5f325cf4897f8c8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:20.572 INFO 4486 --- [ocp-schedules-18,234b2b0cf6ca1691,4366a1f29f92b192] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:41:20.996 INFO 4486 --- [prometheus-cache-monitor1,b491c69ca58ff9b7,2cccd7a12356ef21] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13801,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3448,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:41:20.996 INFO 4486 --- [prometheus-cache-monitor1,b491c69ca58ff9b7,2cccd7a12356ef21] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:41:21.149 INFO 4486 --- [ocp-async-5,0c652b54660e3ca6,06f202408d832ffa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:21.162 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4e301b79f02f02fe,9da2649c8fab16e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4e301b79f02f02fe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:21.163 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4e301b79f02f02fe,9da2649c8fab16e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4e301b79f02f02fe, duration=1 ms] 2024-10-16 13:41:21.483 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1eccc37e4fd4111e,b0c196e82aee88b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1eccc37e4fd4111e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:21.483 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1eccc37e4fd4111e,b0c196e82aee88b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1eccc37e4fd4111e, duration=1 ms] 2024-10-16 13:41:21.892 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:22.994 INFO 4486 --- [prometheus-minute-schedule-1,0b6409b8298705e0,c9351401a71c6b51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:22.999 INFO 4486 --- [ocp-async-5,8a9e7886ed4891a7,0a336bad27722bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:41:23.000 INFO 4486 --- [prometheus-minute-schedule-1,0b6409b8298705e0,c9351401a71c6b51] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:23.063 INFO 4486 --- [ocp-schedules-25,ceafec1a207c2892,c3792192213b67bb] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:41:23.149 INFO 4486 --- [ocp-async-4,90088313dd3b7aec,80db15916add27cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:23.508 INFO 4486 --- [ocp-schedules-17,02d88d555ec33d74,ba64bee0d7f9afa8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:41:23.510 INFO 4486 --- [ocp-schedules-17,02d88d555ec33d74,ba64bee0d7f9afa8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:23.511 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:41:23.511 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:41:23.511 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:41:24.494 INFO 4486 --- [ocp-schedules-30,c25f9b2588a3f0a8,cbc767933607f353] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:41:24.894 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:25.149 INFO 4486 --- [ocp-async-1,25972264a06bd5c6,9b747671c172b002] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:25.150 INFO 4486 --- [ocp-schedules-9,2ec179dae856f784,55ce5ccd618f55b9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:41:25.150 INFO 4486 --- [ocp-schedules-9,2ec179dae856f784,55ce5ccd618f55b9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:41:25.184 INFO 4486 --- [Collect all tenants major compactions1,2ec179dae856f784,ddc0986d9ba88fa0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:41:25.184 INFO 4486 --- [Collect all tenants major compactions2,2ec179dae856f784,d46bdad983636187] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:41:25.184 INFO 4486 --- [Collect all tenants major compactions5,2ec179dae856f784,5163193dd0c4aca1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:41:25.184 INFO 4486 --- [Collect all tenants major compactions4,2ec179dae856f784,a7796018f7885316] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:41:25.185 INFO 4486 --- [Collect all tenants major compactions3,2ec179dae856f784,d34de3cc83c9180a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:41:25.205 INFO 4486 --- [Collect all tenants major compactions1,2ec179dae856f784,ddc0986d9ba88fa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:25.205 INFO 4486 --- [Collect all tenants major compactions2,2ec179dae856f784,d46bdad983636187] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:25.205 INFO 4486 --- [Collect all tenants major compactions3,2ec179dae856f784,d34de3cc83c9180a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:25.206 INFO 4486 --- [Collect all tenants major compactions1,2ec179dae856f784,ddc0986d9ba88fa0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:41:25.207 INFO 4486 --- [Collect all tenants major compactions3,2ec179dae856f784,d34de3cc83c9180a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:41:25.208 INFO 4486 --- [Collect all tenants major compactions2,2ec179dae856f784,d46bdad983636187] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:41:25.210 INFO 4486 --- [Collect all tenants major compactions1,2ec179dae856f784,ddc0986d9ba88fa0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:41:25.210 INFO 4486 --- [Collect all tenants major compactions2,2ec179dae856f784,d46bdad983636187] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:41:25.215 INFO 4486 --- [Collect all tenants major compactions1,2ec179dae856f784,ddc0986d9ba88fa0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:41:25.216 INFO 4486 --- [Collect all tenants major compactions2,2ec179dae856f784,d46bdad983636187] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:41:25.216 INFO 4486 --- [Collect all tenants major compactions4,2ec179dae856f784,a7796018f7885316] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:25.217 INFO 4486 --- [Collect all tenants major compactions5,2ec179dae856f784,5163193dd0c4aca1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:25.218 INFO 4486 --- [Collect all tenants major compactions4,2ec179dae856f784,a7796018f7885316] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:41:25.219 INFO 4486 --- [Collect all tenants major compactions3,2ec179dae856f784,d34de3cc83c9180a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:41:25.219 INFO 4486 --- [Collect all tenants major compactions3,2ec179dae856f784,d34de3cc83c9180a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:41:25.219 INFO 4486 --- [Collect all tenants major compactions5,2ec179dae856f784,5163193dd0c4aca1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:41:25.221 INFO 4486 --- [Collect all tenants major compactions4,2ec179dae856f784,a7796018f7885316] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:41:25.221 INFO 4486 --- [Collect all tenants major compactions4,2ec179dae856f784,a7796018f7885316] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:41:25.225 INFO 4486 --- [Collect all tenants major compactions5,2ec179dae856f784,5163193dd0c4aca1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:41:25.225 INFO 4486 --- [Collect all tenants major compactions5,2ec179dae856f784,5163193dd0c4aca1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:41:25.226 INFO 4486 --- [ocp-schedules-9,2ec179dae856f784,55ce5ccd618f55b9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:41:25.461 INFO 4486 --- [ocp-schedules-10,8e9020d2a3dc2666,425b4bc3df08c1f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:41:25.994 INFO 4486 --- [prometheus-minute-schedule-1,87a618dc61dfbfd4,304081a99ca0abb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:26.001 INFO 4486 --- [prometheus-minute-schedule-1,87a618dc61dfbfd4,304081a99ca0abb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:27.149 INFO 4486 --- [ocp-async-9,df50782b89c59f58,ca26e3728c6948c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:27.161 INFO 4486 --- [ocp-schedules-6,2a033c841a80738a,1036986fe78be869] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:41:27.161+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:41:27.163 INFO 4486 --- [ocp-schedules-6,2a033c841a80738a,1036986fe78be869] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:41:27.163 INFO 4486 --- [ocp-schedules-6,2a033c841a80738a,1036986fe78be869] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:41:27.164 INFO 4486 --- [ocp-schedules-6,2a033c841a80738a,1036986fe78be869] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:41:27.439 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c414eaad01e684c2,6b0a548730de0c7c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c414eaad01e684c2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:27.440 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c414eaad01e684c2,6b0a548730de0c7c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c414eaad01e684c2, duration=3 ms] 2024-10-16 13:41:27.896 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:28.694 INFO 4486 --- [ocp-schedules-3,9be1a9875366d584,c69f4c1c430d2a47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:41:28.742 WARN 4486 --- [scanner-13269532-1,9be1a9875366d584,929be3e66a4dc527] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:41:28.801 INFO 4486 --- [alarm-detect-18,9be1a9875366d584,ffce4cd47267a356] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057260 ~ 1729057278 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057275, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057275, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057275=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:41:28.994 INFO 4486 --- [prometheus-minute-schedule-1,f3080f3a2b906eee,b3971dcab3787462] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:29.000 INFO 4486 --- [prometheus-minute-schedule-1,f3080f3a2b906eee,b3971dcab3787462] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:29.150 INFO 4486 --- [ocp-async-5,b90a4c361092fdce,f19ed22fd29c6db2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:29.605 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7179fd19ad0e02cd,cf65d1484a5d6386] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7179fd19ad0e02cd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:29.605 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,7179fd19ad0e02cd,cf65d1484a5d6386] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7179fd19ad0e02cd, duration=1 ms] 2024-10-16 13:41:30.104 INFO 4486 --- [prometheus-second-schedule-1,2399d3b7d7cdc858,e42f6dc4ee27359f] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:41:30.577 INFO 4486 --- [ocp-schedules-7,46ae33a64001d63f,0e9293430727865a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:41:30.897 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:31.150 INFO 4486 --- [ocp-async-3,69b988968828f4f5,6f9d56b21e015e40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:31.167 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b0b3af905c5c92d4,862aae8c39155492] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b0b3af905c5c92d4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:31.167 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b0b3af905c5c92d4,862aae8c39155492] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b0b3af905c5c92d4, duration=1 ms] 2024-10-16 13:41:31.487 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c34c488207ed767d,6ee6e0b5e88492ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c34c488207ed767d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:31.487 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c34c488207ed767d,6ee6e0b5e88492ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c34c488207ed767d, duration=1 ms] 2024-10-16 13:41:31.994 INFO 4486 --- [prometheus-minute-schedule-1,7bbf948a7c20413a,4e7f4c3560fb1295] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:31.999 INFO 4486 --- [prometheus-minute-schedule-1,7bbf948a7c20413a,4e7f4c3560fb1295] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:32.999 INFO 4486 --- [ocp-async-1,7146fdece4ecb28e,c9deb89544235703] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:41:33.150 INFO 4486 --- [ocp-async-7,dc7eb333e9c95bb8,d4ae1f35dbcdbb21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:33.511 INFO 4486 --- [ocp-schedules-17,02d88d555ec33d74,ba64bee0d7f9afa8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:41:33.898 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:34.994 INFO 4486 --- [prometheus-minute-schedule-1,c6c977dec1804ca6,39c4a622ff11e27e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:35.001 INFO 4486 --- [prometheus-minute-schedule-1,c6c977dec1804ca6,39c4a622ff11e27e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:35.151 INFO 4486 --- [ocp-async-8,59cd81a92fa5a483,59b65555bc942274] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:36.899 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:37.063 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:41:37.065 INFO 4486 --- [ocp-schedules-7,76601e85198add57,ff4449aa0691c1eb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:41:37.066 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:41:37.067 INFO 4486 --- [ocp-schedules-13,24c0500f54187e88,d32bf0211cdaa93d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:41:37.069 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:41:37.069 INFO 4486 --- [ocp-schedules-13,24c0500f54187e88,d32bf0211cdaa93d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:37.071 INFO 4486 --- [ocp-schedules-7,76601e85198add57,ff4449aa0691c1eb] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:41:37.081 INFO 4486 --- [ocp-schedules-11,4e8ca7cc30b1ab72,33f2bfe201e19911] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:41:37.083 INFO 4486 --- [ocp-schedules-11,4e8ca7cc30b1ab72,33f2bfe201e19911] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:37.086 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:41:37.087 INFO 4486 --- [ocp-schedules-21,db9df5e2633e28a2,e0fb7991903479d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 13:41:37.088 INFO 4486 --- [ocp-schedules-21,db9df5e2633e28a2,e0fb7991903479d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:37.095 INFO 4486 --- [ocp-schedules-9,176eb1652e1b7dd3,592a7d09866625bf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:41:37.095 INFO 4486 --- [ocp-schedules-26,792d7b1fdd9f59c7,41ed805b424695cf] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:41:37.096 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:41:37.097 INFO 4486 --- [ocp-schedules-22,ca4d08f4aa47c26e,c975ccae6961512b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:41:37.097 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:41:37.098 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:41:37.098 INFO 4486 --- [host-check-clockdiff1,792d7b1fdd9f59c7,13450eaddb711e2f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:41:37.099 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:41:37.100 INFO 4486 --- [host-check-agent-status1,ca4d08f4aa47c26e,6115a58bdedf6dea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:41:37.101 INFO 4486 --- [host-check-clockdiff2,792d7b1fdd9f59c7,de94231105dc90f7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:41:37.101 INFO 4486 --- [host-check-agent-status2,ca4d08f4aa47c26e,5b719e40ad3405b5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:41:37.102 INFO 4486 --- [host-check-agent-status3,ca4d08f4aa47c26e,9b4e1e0c43855973] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:41:37.102 INFO 4486 --- [host-check-clockdiff3,792d7b1fdd9f59c7,57c0f8eeff7e0d68] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:41:37.105 INFO 4486 --- [host-check-clockdiff4,792d7b1fdd9f59c7,b1e855f0b4226e0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:41:37.105 INFO 4486 --- [host-check-agent-status4,ca4d08f4aa47c26e,a4d68a6df57d4556] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:41:37.109 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:41:37.110 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:37.110 INFO 4486 --- [host-check-agent-status2,ca4d08f4aa47c26e,5b719e40ad3405b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:41:37.110 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:41:37.110 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:41:37.112 INFO 4486 --- [host-check-agent-status1,ca4d08f4aa47c26e,6115a58bdedf6dea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:37.113 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:41:37.113 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:41:37.113 INFO 4486 --- [host-check-agent-status2,ca4d08f4aa47c26e,5b719e40ad3405b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:41:37.113 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:41:37.113 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:37.113 INFO 4486 --- [host-check-agent-status3,ca4d08f4aa47c26e,9b4e1e0c43855973] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:41:37.140 INFO 4486 --- [host-check-agent-status3,ca4d08f4aa47c26e,9b4e1e0c43855973] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:41:37.140 INFO 4486 --- [host-check-agent-status1,ca4d08f4aa47c26e,6115a58bdedf6dea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:37.140 INFO 4486 --- [host-check-agent-status4,ca4d08f4aa47c26e,a4d68a6df57d4556] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:41:37.168 INFO 4486 --- [ocp-async-3,da745a039808f6f4,70ecae5cc6776860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:37.169 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.169 INFO 4486 --- [host-check-clockdiff1,792d7b1fdd9f59c7,13450eaddb711e2f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057297 0 0 , error= 2024-10-16 13:41:37.175 INFO 4486 --- [host-check-agent-status4,ca4d08f4aa47c26e,a4d68a6df57d4556] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:41:37.178 INFO 4486 --- [host-check-clockdiff4,792d7b1fdd9f59c7,b1e855f0b4226e0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057297 3 3 , error= 2024-10-16 13:41:37.176 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:37.181 INFO 4486 --- [host-check-clockdiff3,792d7b1fdd9f59c7,57c0f8eeff7e0d68] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057297 3 3 , error= 2024-10-16 13:41:37.181 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:41:37.183 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.184 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:41:37.202 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,08dc1df015c07304,c82fdf392f96b392] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=08dc1df015c07304, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 13:41:37.203 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,08dc1df015c07304,c82fdf392f96b392] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=08dc1df015c07304, duration=4 ms] 2024-10-16 13:41:37.205 INFO 4486 --- [host-check-agent-status3,ca4d08f4aa47c26e,9b4e1e0c43855973] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:41:37.206 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.206 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:41:37.210 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.210 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:41:37.210 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.210 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:41:37.210 INFO 4486 --- [host-check-agent-status1,ca4d08f4aa47c26e,6115a58bdedf6dea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:41:37.211 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:41:37.211 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:37.212 INFO 4486 --- [host-check-agent-status2,ca4d08f4aa47c26e,5b719e40ad3405b5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:41:37.213 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:41:37.213 INFO 4486 --- [host-check-agent-status4,ca4d08f4aa47c26e,a4d68a6df57d4556] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:41:37.214 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:41:37.215778752+08:00 2024-10-16 13:41:37.214 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:41:37.213308816+08:00 2024-10-16 13:41:37.214 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:41:37.214 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.214 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:41:37.214+08:00 2024-10-16 13:41:37.214 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:41:37.214+08:00 2024-10-16 13:41:37.215 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:41:37.218686920+08:00 2024-10-16 13:41:37.215 INFO 4486 --- [host-check-agent-status3,ca4d08f4aa47c26e,9b4e1e0c43855973] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:41:37.215 INFO 4486 --- [host-check-agent-status3,ca4d08f4aa47c26e,9b4e1e0c43855973] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:41:37.215 INFO 4486 --- [host-check-agent-status3,ca4d08f4aa47c26e,9b4e1e0c43855973] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:41:37.216 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:41:37.216 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:41:37.215+08:00 2024-10-16 13:41:37.216 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:41:37.219776556+08:00 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:41:37.216+08:00 2024-10-16 13:41:37.217 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-agent-status2,ca4d08f4aa47c26e,5b719e40ad3405b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-agent-status2,ca4d08f4aa47c26e,5b719e40ad3405b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-agent-status2,ca4d08f4aa47c26e,5b719e40ad3405b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-agent-status1,ca4d08f4aa47c26e,6115a58bdedf6dea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:41:37.214+08:00 and status AVAILABLE 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-agent-status1,ca4d08f4aa47c26e,6115a58bdedf6dea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-agent-status1,ca4d08f4aa47c26e,6115a58bdedf6dea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:41:37.217 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:41:37.214+08:00 and status AVAILABLE 2024-10-16 13:41:37.218 INFO 4486 --- [host-check-agent-status4,ca4d08f4aa47c26e,a4d68a6df57d4556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:41:37.218 INFO 4486 --- [host-check-agent-status4,ca4d08f4aa47c26e,a4d68a6df57d4556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:41:37.219 INFO 4486 --- [host-check-agent-status4,ca4d08f4aa47c26e,a4d68a6df57d4556] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:41:37.219 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:41:37.220 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:41:37.215+08:00 and status AVAILABLE 2024-10-16 13:41:37.220 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:41:37.216+08:00 and status AVAILABLE 2024-10-16 13:41:37.220 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:41:37.221 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.223 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:41:37.223 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:41:37.223 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.224 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:41:37.225 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:41:37.225 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.226 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.227 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:37.227 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:41:37.228 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:37.229 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:41:37.229 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.230 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:41:37.230 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:41:37.232 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:41:37.232 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:41:37.233 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:41:37.233 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.233 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:41:37.234 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:41:37.238 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.238 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.238 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.238 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.238 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.239 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.243 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.245 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.246 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:41:37.248 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 13:41:37.251 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:37.251 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:37.251 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.255 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:37.255 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:41:37.255 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:41:37.256 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 13:41:37.257 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.258 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:41:37.260 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.260 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.262 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:41:37.264 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 13:41:37.272 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.273 INFO 4486 --- [ocp-schedules-10,8e825d0b4a503f76,4087ac25d01a8b27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:41:37.285 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:41:37.286 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:41:37.290 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:41:37.297 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:41:37.299 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:41:37.302 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:41:37.305 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:41:37.313 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:37.314 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:37.318 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:41:37.320 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:37.333 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:41:37.342 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:41:37.348 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:41:37.352 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:41:37.358 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:41:37.367 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:41:37.370 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:37.377 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:37.380 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:41:37.398 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:41:37.400 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:37.406 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:41:37.406 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:41:37.409 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:41:37.416 INFO 4486 --- [host-check-host-status3,176eb1652e1b7dd3,e53ebef1760a1aa2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:41:37.417 INFO 4486 --- [host-check-host-status2,176eb1652e1b7dd3,da27aa059895d9c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:41:37.425 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:41:37.432 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:41:37.444 INFO 4486 --- [host-check-host-status4,176eb1652e1b7dd3,9670d85e4ae047ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:41:37.447 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,92ab34e60da05a75,23e0faebdd721902] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=92ab34e60da05a75, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:37.447 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,92ab34e60da05a75,23e0faebdd721902] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=92ab34e60da05a75, duration=2 ms] 2024-10-16 13:41:37.480 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:41:37.491 INFO 4486 --- [host-check-host-status1,176eb1652e1b7dd3,c7f90585fd374d93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:41:37.494 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:37.496 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.498 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:41:37.499 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.500 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.501 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:41:37.513 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.513 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.513 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.513 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.513 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.514 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.548 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:41:37.585 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:41:37.635 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:37.650 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:37.651 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.652 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:41:37.653 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:41:37.659 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.660 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:41:37.661 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.661 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.662 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:41:37.672 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.672 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.672 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.672 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.672 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.672 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.679 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:41:37.873 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:41:37.923 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:37.926 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.927 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:41:37.928 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.929 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.930 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:41:37.940 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.940 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.940 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.940 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.940 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:37.941 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:37.958 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:41:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:41:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:41:37.994 INFO 4486 --- [prometheus-minute-schedule-1,79e131e239ca0173,4328e12d15fcdf87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:37.996 INFO 4486 --- [ocp-schedules-30,203a5f00ef3f7111,a1de00b316d62cc4] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 13:41:37.996 INFO 4486 --- [ocp-schedules-30,203a5f00ef3f7111,a1de00b316d62cc4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 13:41:38.001 INFO 4486 --- [prometheus-minute-schedule-1,79e131e239ca0173,4328e12d15fcdf87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:38.034 INFO 4486 --- [pool-10-thread-4,9be9c98cdaf1f6ce,a827d38aa3daf67a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 13:41:38.035 INFO 4486 --- [pool-10-thread-2,9be9c98cdaf1f6ce,d3c067070c7c25bd] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 13:41:38.037 INFO 4486 --- [pool-10-thread-3,9be9c98cdaf1f6ce,6ed6d5e58f529317] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 13:41:38.037 INFO 4486 --- [pool-10-thread-1,9be9c98cdaf1f6ce,4fde0d02b0b51d76] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 13:41:38.044 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 13:41:38.044 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 13:41:38.044 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 13:41:38.044 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 13:41:38.044 INFO 4486 --- [ForkJoinPool-6-worker-1,9be9c98cdaf1f6ce,567ed0a95454354d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 13:41:38.045 INFO 4486 --- [ForkJoinPool-6-worker-1,9be9c98cdaf1f6ce,567ed0a95454354d] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 13:41:38.051 INFO 4486 --- [pool-overview-container-4,7a5fe3b01e7670f6,f4090c7a55a5b7a5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.054 INFO 4486 --- [pool-overview-container-4,7a5fe3b01e7670f6,f4090c7a55a5b7a5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:41:38.060 INFO 4486 --- [pool-10-thread-2,9be9c98cdaf1f6ce,d3c067070c7c25bd] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 13:41:38.060 INFO 4486 --- [pool-10-thread-2,9be9c98cdaf1f6ce,d3c067070c7c25bd] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 13:41:38.067 INFO 4486 --- [host-check-clockdiff2,792d7b1fdd9f59c7,de94231105dc90f7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057298 3 3 , error= 2024-10-16 13:41:38.076 INFO 4486 --- [ForkJoinPool-6-worker-1,9be9c98cdaf1f6ce,567ed0a95454354d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.077 INFO 4486 --- [ForkJoinPool-6-worker-1,9be9c98cdaf1f6ce,567ed0a95454354d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:41:38.079 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.079 INFO 4486 --- [ForkJoinPool-6-worker-1,9be9c98cdaf1f6ce,567ed0a95454354d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.080 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:41:38.080 INFO 4486 --- [ForkJoinPool-6-worker-1,9be9c98cdaf1f6ce,567ed0a95454354d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:41:38.083 INFO 4486 --- [pool-10-thread-1,9be9c98cdaf1f6ce,4fde0d02b0b51d76] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 13:41:38.088 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.092 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:41:38.099 INFO 4486 --- [pool-overview-container-4,7a5fe3b01e7670f6,f4090c7a55a5b7a5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.101 INFO 4486 --- [pool-overview-container-4,7a5fe3b01e7670f6,f4090c7a55a5b7a5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:41:38.104 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.105 WARN 4486 --- [ForkJoinPool-6-worker-1,9be9c98cdaf1f6ce,567ed0a95454354d] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:41:38.105 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:41:38.110 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.111 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:41:38.111 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.116 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:41:38.122 INFO 4486 --- [ForkJoinPool-6-worker-12,9be9c98cdaf1f6ce,a1274b57f39cb1ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.124 INFO 4486 --- [ForkJoinPool-6-worker-12,9be9c98cdaf1f6ce,a1274b57f39cb1ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:41:38.125 INFO 4486 --- [pool-10-thread-1,9be9c98cdaf1f6ce,4fde0d02b0b51d76] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 13:41:38.130 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.132 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:41:38.132 INFO 4486 --- [pool-10-thread-1,9be9c98cdaf1f6ce,4fde0d02b0b51d76] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 13:41:38.135 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.136 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:41:38.137 INFO 4486 --- [ForkJoinPool-6-worker-12,9be9c98cdaf1f6ce,a1274b57f39cb1ca] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 13:41:38.137 INFO 4486 --- [ForkJoinPool-6-worker-12,9be9c98cdaf1f6ce,a1274b57f39cb1ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.139 INFO 4486 --- [ForkJoinPool-6-worker-12,9be9c98cdaf1f6ce,a1274b57f39cb1ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:41:38.140 INFO 4486 --- [ForkJoinPool-6-worker-12,9be9c98cdaf1f6ce,a1274b57f39cb1ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.140 INFO 4486 --- [ForkJoinPool-6-worker-5,9be9c98cdaf1f6ce,c70e8de5b3edea24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.141 INFO 4486 --- [ForkJoinPool-6-worker-12,9be9c98cdaf1f6ce,a1274b57f39cb1ca] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:41:38.142 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.143 INFO 4486 --- [ForkJoinPool-6-worker-5,9be9c98cdaf1f6ce,c70e8de5b3edea24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:41:38.145 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:41:38.152 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:41:38.152 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:38.154 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.156 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:41:38.157 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.158 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.159 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:41:38.161 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.162 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:41:38.162 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:41:38.164 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.165 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:41:38.177 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.177 INFO 4486 --- [ForkJoinPool-6-worker-5,9be9c98cdaf1f6ce,c70e8de5b3edea24] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:41:38.178 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.178 INFO 4486 --- [ForkJoinPool-6-worker-5,9be9c98cdaf1f6ce,c70e8de5b3edea24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.180 INFO 4486 --- [ForkJoinPool-6-worker-5,9be9c98cdaf1f6ce,c70e8de5b3edea24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:41:38.182 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:41:38.183 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:41:38.185 INFO 4486 --- [ForkJoinPool-6-worker-5,9be9c98cdaf1f6ce,c70e8de5b3edea24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.186 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.187 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:41:38.187 INFO 4486 --- [ForkJoinPool-6-worker-5,9be9c98cdaf1f6ce,c70e8de5b3edea24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:41:38.198 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.198 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.199 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:41:38.201 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:41:38.204 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 13:41:38.216 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.217 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:41:38.220 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:41:38.221 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:41:38.221 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.223 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.223 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:41:38.223 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:41:38.224 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.226 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:41:38.229 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 13:41:38.230 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.232 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.232 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:41:38, 2024-10-16 13:41:38] 2024-10-16 13:41:38.232 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:41:38.234 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.238 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 13:41:38.242 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:41:38.244 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:38.246 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.251 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:41:38.252 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:41:38.253 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.254 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:41:38, 2024-10-16 13:41:38] 2024-10-16 13:41:38.259 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.260 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 13:41:38.261 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.261 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:41:38.263 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:41:38, 2024-10-16 13:41:38] 2024-10-16 13:41:38.274 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:41:38.276 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.278 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:41:38, 2024-10-16 13:41:38] 2024-10-16 13:41:38.280 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 13:41:38.280 INFO 4486 --- [ForkJoinPool-6-worker-8,9be9c98cdaf1f6ce,a2090401b9629801] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 13:41:38.280 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:41:38.282 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.284 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:41:38.286 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:38.286 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.286 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:38.287 INFO 4486 --- [ForkJoinPool-6-worker-3,9be9c98cdaf1f6ce,bfa9d249f1136453] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:41:38.287 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.288 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.289 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:41:38.290 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:38.290 INFO 4486 --- [pool-10-thread-4,9be9c98cdaf1f6ce,a827d38aa3daf67a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 13:41:38.290 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:38.291 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:41:38.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.293 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:41:38.297 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 13:41:38.299 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 13:41:38.299 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:38.299 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:38.299 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:38.299 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:38.299 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:41:38.299 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.300 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.302 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:41:38, 2024-10-16 13:41:38] 2024-10-16 13:41:38.330 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:38.330 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:41:38.331 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.331 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:38.333 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:41:38, 2024-10-16 13:41:38] 2024-10-16 13:41:38.336 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 13:41:38.336 INFO 4486 --- [ForkJoinPool-6-worker-10,9be9c98cdaf1f6ce,b48da605fa4fdd20] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 13:41:38.339 INFO 4486 --- [pool-10-thread-3,9be9c98cdaf1f6ce,6ed6d5e58f529317] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 13:41:38.339 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:41:38.363 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:41:38.441 INFO 4486 --- [ocp-schedules-19,1d01e1853e3bbd26,747f7d2394566737] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:41:38.441 INFO 4486 --- [ocp-schedules-19,1d01e1853e3bbd26,747f7d2394566737] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:41:38.442 INFO 4486 --- [ocp-schedules-19,1d01e1853e3bbd26,747f7d2394566737] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:41:38.469 INFO 4486 --- [ocp-schedules-24,3f959b42bad55b47,fbdcd6b343eca2f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13801,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:41:38.470 INFO 4486 --- [ocp-schedules-24,3f959b42bad55b47,fbdcd6b343eca2f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:41:38.605 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:41:38.366173, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:41:38.605 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:41:38.366173, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:41:38.645 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:41:38.366173} } 2024-10-16 13:41:38.711 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:41:38.720 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:41:38.724 INFO 4486 --- [ocp-schedules-17,eef9cdc9a406e429,e3539699280aec05] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:41:38.830 INFO 4486 --- [ocp-schedules-15,8333fa47ea364f5e,64d53daade380305] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:41:38.930 WARN 4486 --- [scanner-13269792-1,8333fa47ea364f5e,fcaf4766f7a93902] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:41:38.969 INFO 4486 --- [ocp-schedules-6,3c0e8b6f0eb4f34c,01dd6db52cced415] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:41:39.168 INFO 4486 --- [ocp-async-7,d9182ab396fe5b38,80a2ef3a93d5b866] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:39.611 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,692e63a13d989fb4,ee3a9104a79158b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=692e63a13d989fb4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:39.611 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,692e63a13d989fb4,ee3a9104a79158b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=692e63a13d989fb4, duration=1 ms] 2024-10-16 13:41:39.901 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:40.006 INFO 4486 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:41:40.007 INFO 4486 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:41:40.027 INFO 4486 --- [Check obproxy2,a0b242f0ec77be4b,a2712021623443b1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:41:40.027 INFO 4486 --- [Check obproxy2,a0b242f0ec77be4b,a2712021623443b1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:40.030 INFO 4486 --- [Check obproxy1,757ee04f5c1312d9,836b550a5f3b4d52] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:41:40.030 INFO 4486 --- [Check obproxy1,757ee04f5c1312d9,836b550a5f3b4d52] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:40.031 INFO 4486 --- [Collect obproxy2,9842106fce35c22d,9a4d7f544da0fb5b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:41:40.031 INFO 4486 --- [Collect obproxy2,9842106fce35c22d,9a4d7f544da0fb5b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:40.033 INFO 4486 --- [Collect obproxy1,ea9aebafc6577a6b,84af361c2735ac14] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:41:40.033 INFO 4486 --- [Collect obproxy1,ea9aebafc6577a6b,84af361c2735ac14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:41:40.053 INFO 4486 --- [Check obproxy2,a0b242f0ec77be4b,a2712021623443b1] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:41:40.053 INFO 4486 --- [Check obproxy2,a0b242f0ec77be4b,a2712021623443b1] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:41:40.054 INFO 4486 --- [Collect obproxy2,9842106fce35c22d,9a4d7f544da0fb5b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:41:40.059 INFO 4486 --- [Collect obproxy1,ea9aebafc6577a6b,84af361c2735ac14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:41:40.060 INFO 4486 --- [Collect obproxy2,9842106fce35c22d,9a4d7f544da0fb5b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:41:40.064 INFO 4486 --- [Check obproxy1,757ee04f5c1312d9,836b550a5f3b4d52] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:41:40.064 INFO 4486 --- [Check obproxy1,757ee04f5c1312d9,836b550a5f3b4d52] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:41:40.064 INFO 4486 --- [Collect obproxy1,ea9aebafc6577a6b,84af361c2735ac14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:41:40.066 INFO 4486 --- [Collect obproxy2,9842106fce35c22d,9a4d7f544da0fb5b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:41:40.069 INFO 4486 --- [Collect obproxy1,ea9aebafc6577a6b,84af361c2735ac14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:41:40.583 INFO 4486 --- [ocp-schedules-26,488620a3f859dbf9,a9d24508ffb060b6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:41:40.994 INFO 4486 --- [prometheus-minute-schedule-1,12eb1375a6fb81d2,e25c09f046a8c5ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:40.995 INFO 4486 --- [prometheus-hour-schedule-1,bb17268cb8ce4db5,99f57f7728f10dbd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:41:40.999 INFO 4486 --- [prometheus-minute-schedule-1,12eb1375a6fb81d2,e25c09f046a8c5ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:41.168 INFO 4486 --- [ocp-async-8,7872fa7faae64280,cb4ded582d76dfc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:41.174 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,3d471183a7de3ba8,3b25ce44adfcaffc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3d471183a7de3ba8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:41.174 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,3d471183a7de3ba8,3b25ce44adfcaffc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3d471183a7de3ba8, duration=1 ms] 2024-10-16 13:41:41.491 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,aefc1b48186209f3,eb80d9a5f4180540] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=aefc1b48186209f3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:41.492 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,aefc1b48186209f3,eb80d9a5f4180540] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=aefc1b48186209f3, duration=2 ms] 2024-10-16 13:41:41.859 INFO 4486 --- [ocp-schedules-16,7efd3d024ab34ea9,dff3770f84abffcf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:41:41.859 INFO 4486 --- [ocp-schedules-16,edf7470635f42fe4,6389f7ffa29e0471] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:41:41.861 INFO 4486 --- [ocp-schedules-16,edf7470635f42fe4,6389f7ffa29e0471] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:41.893 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:41.893 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:41.901 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:41.901 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:41.926 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:41.927 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:41.928 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:41:41.929 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:41.930 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:41.957 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:41.958 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:41.958 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:41.959 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:41:41.959 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:41.963 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:41:41.964 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:41.972 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:41.972 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:41.985 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:41.994 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:41.996 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:41.997 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:41.999 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:41:42.004 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:41:42.010 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:42.012 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:42.034 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:42.036 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:42.037 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:41:42.040 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:41:42.063 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:41:42.063 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:41:42.092 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:42.094 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:42.095 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:41:42.098 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:42.101 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:41:42.104 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:42.110 INFO 4486 --- [ocp-schedules-19,78addcce5d64c887,1bd75acd67414188] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:41:42.117 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:41:42.126 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:42.127 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:42.128 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:41:42.129 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:42.129 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:42.136 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:41:42.136 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:41:42.157 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:42.157 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:42.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:42.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:42.214 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:42.214 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:41:42.241 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:42.241 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:41:42.268 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:41:42.293 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:42.295 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:42.296 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:41:42.300 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:41:42.305 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:41:42.310 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:41:42.312 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:41:42.333 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:41:42.476 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:41:42.502 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:41:42.503 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:42.504 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:41:42.532 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:41:42.532 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:41:42.902 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:43.000 INFO 4486 --- [ocp-async-8,196bedadfa4dab24,54cdcac3b2d186c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:41:43.168 INFO 4486 --- [ocp-async-2,36acb790deebb053,e3f7c7a02a762aac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:43.994 INFO 4486 --- [prometheus-minute-schedule-1,5eed600b050e1aa0,7f8367645268c512] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:44.000 INFO 4486 --- [prometheus-minute-schedule-1,5eed600b050e1aa0,7f8367645268c512] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:44.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,c6aa59526e667859,582c05af82912b78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=c6aa59526e667859, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:41:44.485 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,c6aa59526e667859,582c05af82912b78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=c6aa59526e667859, duration=2 ms] 2024-10-16 13:41:45.169 INFO 4486 --- [ocp-async-6,b4ff5ace9a1c747e,9e7e164e2f3a87f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:45.486 INFO 4486 --- [ocp-schedules-22,b3acd9bf2ee91290,30ecbe86e12a9cb9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:41:45.492 INFO 4486 --- [ocp-schedules-22,b3acd9bf2ee91290,30ecbe86e12a9cb9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:41:45.492 INFO 4486 --- [ocp-schedules-22,b3acd9bf2ee91290,30ecbe86e12a9cb9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:41:45.507 INFO 4486 --- [ocp-schedules-3,cd0807434c1577e7,a706125af7e97d7e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:41:45.507 INFO 4486 --- [ocp-schedules-3,cd0807434c1577e7,a706125af7e97d7e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:41:45.902 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:46.994 INFO 4486 --- [prometheus-minute-schedule-1,f2a90fccfa897a31,58485f2cd7a2acea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:47.000 INFO 4486 --- [prometheus-minute-schedule-1,f2a90fccfa897a31,58485f2cd7a2acea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:47.013 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,7a915aee22ac1b9e,f53e67a20d6d7f64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=7a915aee22ac1b9e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:41:47.013 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,7a915aee22ac1b9e,f53e67a20d6d7f64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=7a915aee22ac1b9e, duration=1 ms] 2024-10-16 13:41:47.071 INFO 4486 --- [ocp-schedules-13,24c0500f54187e88,d32bf0211cdaa93d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:41:47.140 INFO 4486 --- [ocp-schedules-30,8231a311c67488f1,1fc203d32f0b963f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:41:47.169 INFO 4486 --- [ocp-async-8,456550f6669f5c9c,60bd4951e7cb9586] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:47.216 INFO 4486 --- [ocp-schedules-11,4e8ca7cc30b1ab72,33f2bfe201e19911] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:41:47.268 INFO 4486 --- [ocp-schedules-21,db9df5e2633e28a2,e0fb7991903479d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 13:41:47.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,616f649c3e4d7869,9ad6b08c4abd0644] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=616f649c3e4d7869, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:41:47.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,616f649c3e4d7869,9ad6b08c4abd0644] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=616f649c3e4d7869, duration=1 ms] 2024-10-16 13:41:47.452 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,aa959c7effcdb576,ae60d55cc3c4eca9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=aa959c7effcdb576, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:47.452 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,aa959c7effcdb576,ae60d55cc3c4eca9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=aa959c7effcdb576, duration=1 ms] 2024-10-16 13:41:48.759 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:48.760 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:41:48.761 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:41:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:41:48.763 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:41:48.903 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:49.080 INFO 4486 --- [ocp-schedules-28,f10666dc8fb4083a,628b0672fa38c624] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:41:49.113 WARN 4486 --- [scanner-13270125-1,f10666dc8fb4083a,accf775836e55359] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:41:49.162 INFO 4486 --- [alarm-detect-17,f10666dc8fb4083a,97ea5d5299dcde51] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057275 ~ 1729057299 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057290, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057290, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057290=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:41:49.169 INFO 4486 --- [ocp-async-3,ee73f4b0a430ff41,20781b40a8a7e7ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:49.616 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,f147b451a57cef65,35b5079d66117842] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f147b451a57cef65, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:49.616 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,f147b451a57cef65,35b5079d66117842] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f147b451a57cef65, duration=1 ms] 2024-10-16 13:41:49.994 INFO 4486 --- [prometheus-minute-schedule-1,1db5af7fb19db0c5,5a687fd722431c91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:50.003 INFO 4486 --- [prometheus-minute-schedule-1,1db5af7fb19db0c5,5a687fd722431c91] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:50.419 INFO 4486 --- [ocp-schedules-24,2347d637b5931d0d,0398d676038460af] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:41:50.419 INFO 4486 --- [ocp-schedules-24,2347d637b5931d0d,0398d676038460af] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:41:50.419 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:41:50.419 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:41:50.419 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:41:50.419 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:41:50.419 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:41:50.422 INFO 4486 --- [ocp-schedules-30,0014d0ced93471c1,bc6beb8716cea991] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:41:50.422 INFO 4486 --- [ocp-schedules-30,0014d0ced93471c1,bc6beb8716cea991] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:41:50.424 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:41:50.425 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:41:50.425 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:41:50.425 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:41:50.425 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:41:50.426 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:41:50.426 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:41:50.426 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:41:50.426 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:41:50.427 INFO 4486 --- [ocp-schedules-24,2347d637b5931d0d,0398d676038460af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:41:50.427 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:41:50.428 INFO 4486 --- [ocp-schedules-30,0014d0ced93471c1,bc6beb8716cea991] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:41:50.428 INFO 4486 --- [ocp-schedules-30,0014d0ced93471c1,bc6beb8716cea991] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:41:50.428 INFO 4486 --- [ocp-schedules-24,2347d637b5931d0d,0398d676038460af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:41:50.432 INFO 4486 --- [ocp-schedules-30,0014d0ced93471c1,bc6beb8716cea991] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:41:50.432 INFO 4486 --- [ocp-schedules-30,0014d0ced93471c1,bc6beb8716cea991] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:41:50.451 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:41:50.452 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:41:50.453 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:41:50.454 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:41:50.454 INFO 4486 --- [ocp-schedules-24,2347d637b5931d0d,0398d676038460af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:41:50.458 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:41:50.461 INFO 4486 --- [ocp-schedules-24,2347d637b5931d0d,0398d676038460af] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:41:50.589 INFO 4486 --- [ocp-schedules-20,094fe64139e31e97,ff1a6d0be9efaa9b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:41:51.170 INFO 4486 --- [ocp-async-7,dfa655b21dedcc53,5e632a65e3584483] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:51.178 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b9fefe415fdc4c26,77bb44680f65c1f4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b9fefe415fdc4c26, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:51.178 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b9fefe415fdc4c26,77bb44680f65c1f4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b9fefe415fdc4c26, duration=1 ms] 2024-10-16 13:41:51.498 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5025f7b3a29d6bb7,540ec44a71695c28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5025f7b3a29d6bb7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:51.498 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5025f7b3a29d6bb7,540ec44a71695c28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5025f7b3a29d6bb7, duration=2 ms] 2024-10-16 13:41:51.905 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:52.994 INFO 4486 --- [prometheus-minute-schedule-1,65014248b3f7383e,3d40bcbb1ac79ba4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:53.000 INFO 4486 --- [ocp-async-7,774bea428397c6c2,018515a3c647368b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:41:53.001 INFO 4486 --- [prometheus-minute-schedule-1,65014248b3f7383e,3d40bcbb1ac79ba4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:53.170 INFO 4486 --- [ocp-async-8,bebdb787ecc03203,5fb1a6c80f23b027] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:54.906 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:55.170 INFO 4486 --- [ocp-async-1,ab521ecfa4cac897,e62249f13dd0c9c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:55.994 INFO 4486 --- [prometheus-minute-schedule-1,52972bf746423ae6,5357b95a7a93a684] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:56.000 INFO 4486 --- [prometheus-minute-schedule-1,52972bf746423ae6,5357b95a7a93a684] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:57.170 INFO 4486 --- [ocp-async-6,7043261f384217d5,2c5f5ba52422a730] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:57.456 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,56aed47f920fb21e,0cedc905aa7e48e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=56aed47f920fb21e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:57.457 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,56aed47f920fb21e,0cedc905aa7e48e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=56aed47f920fb21e, duration=2 ms] 2024-10-16 13:41:57.908 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:41:58.728 INFO 4486 --- [ocp-schedules-21,019013ba2fb240bb,31a6375dc707ad44] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:41:58.731 INFO 4486 --- [ocp-schedules-21,019013ba2fb240bb,31a6375dc707ad44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:41:58.976 INFO 4486 --- [ocp-schedules-15,5288683fbf84bce9,1a349f35a6977c25] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:41:58.994 INFO 4486 --- [prometheus-minute-schedule-1,2e2ec6ad1fe07779,1d88391b529c8c96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057320, interval=60 2024-10-16 13:41:59.000 INFO 4486 --- [prometheus-minute-schedule-1,2e2ec6ad1fe07779,1d88391b529c8c96] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:41:59.170 INFO 4486 --- [ocp-async-5,f8077add42d19d92,f977ae9acb8edeef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:41:59.217 INFO 4486 --- [ocp-schedules-17,b66a5a515197385c,19a6108dee86eebb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:41:59.248 WARN 4486 --- [scanner-13270345-1,b66a5a515197385c,4ceee0084e809813] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:41:59.338 INFO 4486 --- [alarm-detect-2,b66a5a515197385c,60717f04fdd2b094] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057290 ~ 1729057309 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057305, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057305, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057305=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:41:59.621 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,6ff3ff1465c0b672,3c86e90599e9f029] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6ff3ff1465c0b672, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:41:59.621 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,6ff3ff1465c0b672,3c86e90599e9f029] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6ff3ff1465c0b672, duration=1 ms] 2024-10-16 13:42:00.595 INFO 4486 --- [ocp-schedules-23,53a5184250e6b7c4,16569da58f804569] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:42:00.909 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:01.170 INFO 4486 --- [ocp-async-8,8dfbe1854dd39d09,6272c1682dd2a2f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:01.185 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,92ae26eb8c8c3c99,a729508201d1f42d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=92ae26eb8c8c3c99, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:01.185 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,92ae26eb8c8c3c99,a729508201d1f42d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=92ae26eb8c8c3c99, duration=1 ms] 2024-10-16 13:42:01.504 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,7bcda57d90283277,5c38f567294676df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7bcda57d90283277, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:01.505 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,7bcda57d90283277,5c38f567294676df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7bcda57d90283277, duration=1 ms] 2024-10-16 13:42:01.567 INFO 4486 --- [ocp-schedules-18,03b432c50d5e2fb3,7ea7c40480597051] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:42:01.567+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:42:01.572 INFO 4486 --- [ocp-schedules-18,03b432c50d5e2fb3,7ea7c40480597051] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:42:01.583 INFO 4486 --- [ocp-schedules-18,03b432c50d5e2fb3,7ea7c40480597051] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:42:01.595 INFO 4486 --- [ocp-schedules-18,03b432c50d5e2fb3,7ea7c40480597051] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:42:01.994 INFO 4486 --- [prometheus-minute-schedule-1,cba918a10343c863,c9ac2f62395b0d5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:01.999 INFO 4486 --- [prometheus-minute-schedule-1,cba918a10343c863,c9ac2f62395b0d5e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:03.000 INFO 4486 --- [ocp-async-8,cf11eae9b0f7bdaa,ff48a36837d5135e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:42:03.171 INFO 4486 --- [ocp-async-9,431e3185a68916bc,0cd1fe49ed440b06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:03.910 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:04.994 INFO 4486 --- [prometheus-minute-schedule-1,c40ea062ad0bb508,b3eb1bf3464662c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:05.001 INFO 4486 --- [prometheus-minute-schedule-1,c40ea062ad0bb508,b3eb1bf3464662c8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:05.171 INFO 4486 --- [ocp-async-4,48afbd0d55a52d17,5f97df20afced0ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:06.911 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:07.066 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:42:07.067 INFO 4486 --- [ocp-schedules-11,9d76125d560acf40,00d066c100d7bb36] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:42:07.069 INFO 4486 --- [ocp-schedules-11,9d76125d560acf40,00d066c100d7bb36] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:07.081 INFO 4486 --- [ocp-schedules-28,57a5b4d3aac41cd3,f2b2594d7a3171ed] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:42:07.084 INFO 4486 --- [ocp-schedules-28,57a5b4d3aac41cd3,f2b2594d7a3171ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:42:07.090 INFO 4486 --- [ocp-schedules-3,d3f1b605bc4ce6ec,c223d3a590f9879e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:42:07.090 INFO 4486 --- [ocp-schedules-24,98bff2e13cf16dd3,532f8518cb87ad29] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:42:07.091 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:42:07.091 INFO 4486 --- [ocp-schedules-23,a592b4d4ed7ebfce,b7d8885fc4338440] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:42:07.091 INFO 4486 --- [host-check-agent-status1,98bff2e13cf16dd3,23dc1cfd20b6fef9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:42:07.092 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:42:07.093 INFO 4486 --- [host-check-clockdiff2,a592b4d4ed7ebfce,6046cfbd5e81a830] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:42:07.093 INFO 4486 --- [host-check-clockdiff1,a592b4d4ed7ebfce,ecfd72115cdf2589] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:42:07.093 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:42:07.094 INFO 4486 --- [host-check-agent-status2,98bff2e13cf16dd3,45e6f55fcf02fe77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:42:07.094 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:42:07.095 INFO 4486 --- [host-check-clockdiff3,a592b4d4ed7ebfce,0a3d55ce145f2370] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:42:07.095 INFO 4486 --- [host-check-agent-status3,98bff2e13cf16dd3,33dead1ec962bebd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:42:07.096 INFO 4486 --- [host-check-clockdiff4,a592b4d4ed7ebfce,854971e53b93b57f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:42:07.096 INFO 4486 --- [host-check-agent-status4,98bff2e13cf16dd3,6a8c738e4f4b040a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:42:07.100 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:07.100 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:07.100 INFO 4486 --- [host-check-agent-status2,98bff2e13cf16dd3,45e6f55fcf02fe77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:07.102 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:07.102 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:07.102 INFO 4486 --- [host-check-agent-status1,98bff2e13cf16dd3,23dc1cfd20b6fef9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:07.102 INFO 4486 --- [host-check-agent-status2,98bff2e13cf16dd3,45e6f55fcf02fe77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:07.102 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:07.110 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:07.110 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:07.136 INFO 4486 --- [host-check-agent-status4,98bff2e13cf16dd3,6a8c738e4f4b040a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:07.111 INFO 4486 --- [host-check-agent-status1,98bff2e13cf16dd3,23dc1cfd20b6fef9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:07.143 INFO 4486 --- [host-check-agent-status3,98bff2e13cf16dd3,33dead1ec962bebd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:07.148 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:07.148 INFO 4486 --- [host-check-clockdiff3,a592b4d4ed7ebfce,0a3d55ce145f2370] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057327 3 3 , error= 2024-10-16 13:42:07.148 INFO 4486 --- [host-check-clockdiff1,a592b4d4ed7ebfce,ecfd72115cdf2589] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057327 0 0 , error= 2024-10-16 13:42:07.148 INFO 4486 --- [host-check-clockdiff2,a592b4d4ed7ebfce,6046cfbd5e81a830] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057327 4 4 , error= 2024-10-16 13:42:07.149 INFO 4486 --- [host-check-agent-status4,98bff2e13cf16dd3,6a8c738e4f4b040a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:07.149 INFO 4486 --- [host-check-agent-status3,98bff2e13cf16dd3,33dead1ec962bebd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:07.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:07.152 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:42:07.171 INFO 4486 --- [ocp-async-8,2d95f5868b758218,4f0c9c7b9ad0698d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:07.174 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:42:07.174 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:42:07.175 INFO 4486 --- [host-check-agent-status2,98bff2e13cf16dd3,45e6f55fcf02fe77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:42:07.177 INFO 4486 --- [host-check-agent-status1,98bff2e13cf16dd3,23dc1cfd20b6fef9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:42:07.178 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:42:07.180854806+08:00 2024-10-16 13:42:07.178 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:42:07.177292595+08:00 2024-10-16 13:42:07.178 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:42:07.178 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:42:07.178+08:00 2024-10-16 13:42:07.178 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:42:07.178+08:00 2024-10-16 13:42:07.179 INFO 4486 --- [host-check-agent-status3,98bff2e13cf16dd3,33dead1ec962bebd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:42:07.179 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.180 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:42:07.181 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:07.181 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:42:07.178+08:00 and status AVAILABLE 2024-10-16 13:42:07.181 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:42:07.184774055+08:00 2024-10-16 13:42:07.181 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:42:07.178+08:00 and status AVAILABLE 2024-10-16 13:42:07.181 INFO 4486 --- [host-check-agent-status2,98bff2e13cf16dd3,45e6f55fcf02fe77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:42:07.182 INFO 4486 --- [host-check-agent-status2,98bff2e13cf16dd3,45e6f55fcf02fe77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:42:07.182 INFO 4486 --- [host-check-agent-status2,98bff2e13cf16dd3,45e6f55fcf02fe77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:42:07.182 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:42:07.182+08:00 2024-10-16 13:42:07.183 INFO 4486 --- [host-check-agent-status4,98bff2e13cf16dd3,6a8c738e4f4b040a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:42:07.183 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:42:07.185613903+08:00 2024-10-16 13:42:07.183 INFO 4486 --- [host-check-agent-status1,98bff2e13cf16dd3,23dc1cfd20b6fef9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:42:07.183 INFO 4486 --- [host-check-agent-status1,98bff2e13cf16dd3,23dc1cfd20b6fef9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:42:07.183 INFO 4486 --- [host-check-agent-status1,98bff2e13cf16dd3,23dc1cfd20b6fef9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:42:07.184 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:42:07.183+08:00 2024-10-16 13:42:07.184 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:42:07.184 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:42:07.185 INFO 4486 --- [host-check-agent-status3,98bff2e13cf16dd3,33dead1ec962bebd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:42:07.185 INFO 4486 --- [host-check-agent-status3,98bff2e13cf16dd3,33dead1ec962bebd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:42:07.185 INFO 4486 --- [host-check-agent-status3,98bff2e13cf16dd3,33dead1ec962bebd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:42:07.186 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:42:07.186 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:42:07.182+08:00 and status AVAILABLE 2024-10-16 13:42:07.187 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:42:07.183+08:00 and status AVAILABLE 2024-10-16 13:42:07.187 INFO 4486 --- [host-check-agent-status4,98bff2e13cf16dd3,6a8c738e4f4b040a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:42:07.187 INFO 4486 --- [host-check-agent-status4,98bff2e13cf16dd3,6a8c738e4f4b040a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:42:07.187 INFO 4486 --- [host-check-agent-status4,98bff2e13cf16dd3,6a8c738e4f4b040a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:42:07.189 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:42:07.189 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:42:07.191 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.193 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:07.194 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:07.195 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:07.196 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:07.196 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:42:07.198 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:07.199 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:07.199 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:07.201 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:07.205 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.206 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:07.216 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.217 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:07.218 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:07.222 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:07.223 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:07.226 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.227 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:07.229 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:07.229 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.230 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:42:07.240 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:07.241 INFO 4486 --- [ocp-schedules-14,6eefff155bb5f826,82662e3ace7ecff0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:42:07.265 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:42:07.271 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:42:07.273 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:42:07.280 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:42:07.281 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:42:07.283 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:42:07.284 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:42:07.292 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:42:07.298 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:07.299 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:07.299 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:07.307 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:07.327 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:42:07.332 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:42:07.337 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:42:07.340 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:42:07.345 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:42:07.347 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:42:07.361 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:07.362 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:42:07.367 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:07.368 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:07.375 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:42:07.391 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:07.392 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:42:07.398 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:42:07.402 INFO 4486 --- [host-check-host-status3,d3f1b605bc4ce6ec,774ae8efc7792136] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:42:07.405 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:42:07.411 INFO 4486 --- [host-check-host-status4,d3f1b605bc4ce6ec,d282ffc242890fca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:42:07.417 INFO 4486 --- [host-check-host-status2,d3f1b605bc4ce6ec,800c0952dd478fcd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:42:07.452 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:42:07.463 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ac6909f1abcba8f9,44de8930f1eae6fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ac6909f1abcba8f9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:07.463 INFO 4486 --- [host-check-host-status1,d3f1b605bc4ce6ec,a6ef800ccdbd0847] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:42:07.463 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ac6909f1abcba8f9,44de8930f1eae6fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ac6909f1abcba8f9, duration=2 ms] 2024-10-16 13:42:07.994 INFO 4486 --- [prometheus-minute-schedule-1,c348e131033fc4e0,5721b1d36e38b34a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:08.003 INFO 4486 --- [prometheus-minute-schedule-1,c348e131033fc4e0,5721b1d36e38b34a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:08.066 INFO 4486 --- [host-check-clockdiff4,a592b4d4ed7ebfce,854971e53b93b57f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057328 3 3 , error= 2024-10-16 13:42:08.762 INFO 4486 --- [ocp-schedules-21,019013ba2fb240bb,31a6375dc707ad44] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:42:09.171 INFO 4486 --- [ocp-async-6,26e27ad660736e79,33e7db1e1618040c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:09.441 INFO 4486 --- [ocp-schedules-24,d178741baf64c312,41bfb54069096032] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:42:09.488 WARN 4486 --- [scanner-13270669-1,d178741baf64c312,b640e038764272ec] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:42:09.626 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,e650024e14183ac0,306caeebfc0ea4e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e650024e14183ac0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:09.626 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,e650024e14183ac0,306caeebfc0ea4e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e650024e14183ac0, duration=1 ms] 2024-10-16 13:42:09.912 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:10.006 INFO 4486 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:42:10.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:42:10.028 INFO 4486 --- [Check obproxy2,96d64b82d3f90896,9b5708c1383b713d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:42:10.028 INFO 4486 --- [Check obproxy1,de3ff636168c24f5,e5d7ec344e5e7e0f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:42:10.028 INFO 4486 --- [Check obproxy2,96d64b82d3f90896,9b5708c1383b713d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:10.028 INFO 4486 --- [Check obproxy1,de3ff636168c24f5,e5d7ec344e5e7e0f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:10.031 INFO 4486 --- [Collect obproxy2,9c08cc0f40e60d28,c6dd028e34594cf8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:42:10.032 INFO 4486 --- [Collect obproxy2,9c08cc0f40e60d28,c6dd028e34594cf8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:10.032 INFO 4486 --- [Collect obproxy1,c7e9f711733b2b42,35ef8efd01dc0969] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:42:10.032 INFO 4486 --- [Collect obproxy1,c7e9f711733b2b42,35ef8efd01dc0969] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:10.053 INFO 4486 --- [Check obproxy2,96d64b82d3f90896,9b5708c1383b713d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:42:10.053 INFO 4486 --- [Check obproxy2,96d64b82d3f90896,9b5708c1383b713d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:42:10.055 INFO 4486 --- [Collect obproxy2,9c08cc0f40e60d28,c6dd028e34594cf8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:42:10.056 INFO 4486 --- [Check obproxy1,de3ff636168c24f5,e5d7ec344e5e7e0f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:42:10.056 INFO 4486 --- [Check obproxy1,de3ff636168c24f5,e5d7ec344e5e7e0f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:42:10.058 INFO 4486 --- [Collect obproxy1,c7e9f711733b2b42,35ef8efd01dc0969] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:42:10.060 INFO 4486 --- [Collect obproxy2,9c08cc0f40e60d28,c6dd028e34594cf8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:42:10.062 INFO 4486 --- [Collect obproxy1,c7e9f711733b2b42,35ef8efd01dc0969] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:42:10.064 INFO 4486 --- [Collect obproxy1,c7e9f711733b2b42,35ef8efd01dc0969] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:42:10.066 INFO 4486 --- [Collect obproxy2,9c08cc0f40e60d28,c6dd028e34594cf8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:42:10.602 INFO 4486 --- [ocp-schedules-9,9f0579351ea0ef77,936613368a71e0b4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:42:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:42:10.994 INFO 4486 --- [prometheus-minute-schedule-1,a94d81c04c459239,e9153dc9a899c769] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:11.001 INFO 4486 --- [prometheus-minute-schedule-1,a94d81c04c459239,e9153dc9a899c769] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:42:11.172 INFO 4486 --- [ocp-async-7,3466a8c1d487831b,4ea8c65bebbaf2a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:11.204 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e4adb6bc2a97d5dc,cde617a8f05122de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e4adb6bc2a97d5dc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:11.204 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e4adb6bc2a97d5dc,cde617a8f05122de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e4adb6bc2a97d5dc, duration=1 ms] 2024-10-16 13:42:11.510 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,1c35cc9e189edff3,4fc99870ade69799] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1c35cc9e189edff3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:11.510 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,1c35cc9e189edff3,4fc99870ade69799] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1c35cc9e189edff3, duration=1 ms] 2024-10-16 13:42:12.533 INFO 4486 --- [ocp-schedules-16,edf7470635f42fe4,6389f7ffa29e0471] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:42:12.533 INFO 4486 --- [ocp-schedules-16,72bfabf8b7f6612a,b5684414948242a1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:42:12.535 INFO 4486 --- [ocp-schedules-16,72bfabf8b7f6612a,b5684414948242a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:12.568 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.568 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:12.583 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.584 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:12.610 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.611 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:12.617 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:12.618 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.618 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:42:12.656 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:12.657 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.658 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:12.659 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:42:12.659 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.662 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:42:12.664 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:12.673 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.673 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:12.702 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.712 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:12.712 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.713 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:12.714 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:42:12.718 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:42:12.722 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:12.724 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:12.731 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:12.733 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.734 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:42:12.749 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:42:12.760 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:42:12.760 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:42:12.785 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:12.786 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.788 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:12.791 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.794 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:42:12.797 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:12.829 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:42:12.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:12.840 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:12.841 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.842 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:42:12.852 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:42:12.852 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:42:12.857 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.857 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:12.882 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.882 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:12.908 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.908 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:12.913 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:12.934 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.934 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:12.960 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:12.986 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:12.987 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:12.988 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:12.992 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:42:12.998 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:42:13.000 INFO 4486 --- [ocp-async-7,add1222ba86da270,329bf3f3d5e3229b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:42:13.003 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:42:13.005 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:42:13.032 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:42:13.172 INFO 4486 --- [ocp-async-2,2c241298ac69ce33,bd381b2b47adf62d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:13.186 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:42:13.212 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:13.214 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:13.216 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:42:13.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:42:13.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:42:13.604 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057260, nodeCount=1000, affectRows=1000 2024-10-16 13:42:13.609 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057260, nodeCount=999, validCount=999 2024-10-16 13:42:13.617 ERROR 4486 --- [metric-archive-second-31,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620709) Duplicate entry '669741696503320576-1729057260' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:42:13.616198] [YB420AC531B8-00060B7DB419624B-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620709) Duplicate entry '669741696503320576-1729057260' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:42:13.616198] [YB420AC531B8-00060B7DB419624B-0-0] 2024-10-16 13:42:13.635 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057260, nodeCount=1000, affectRows=1000 2024-10-16 13:42:13.639 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057260, nodeCount=1000, validCount=1000 2024-10-16 13:42:13.654 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057260, nodeCount=1000, affectRows=1000 2024-10-16 13:42:13.657 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057260, nodeCount=1000, validCount=1000 2024-10-16 13:42:13.840 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057260, nodeCount=449, affectRows=449 2024-10-16 13:42:13.841 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057260, nodeCount=447, validCount=447 2024-10-16 13:42:13.994 INFO 4486 --- [prometheus-minute-schedule-1,02dad6f92419a4c1,4078051e95f00e4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:14.001 INFO 4486 --- [prometheus-minute-schedule-1,02dad6f92419a4c1,4078051e95f00e4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:14.660 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057260, nodeCount=44, affectRows=44 2024-10-16 13:42:14.662 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057260, nodeCount=46, affectRows=46 2024-10-16 13:42:14.663 INFO 4486 --- [metric-archive-async-16,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057260, nodeCount=124, affectRows=124 2024-10-16 13:42:14.665 INFO 4486 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057260, nodeCount=122, affectRows=122 2024-10-16 13:42:14.666 INFO 4486 --- [metric-archive-async-21,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057260, nodeCount=141, affectRows=141 2024-10-16 13:42:14.738 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057260, nodeCount=46, validCount=46 2024-10-16 13:42:14.743 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057260, nodeCount=44, validCount=44 2024-10-16 13:42:14.859 INFO 4486 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057260, nodeCount=122, validCount=122 2024-10-16 13:42:14.862 INFO 4486 --- [metric-archive-async-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057260, nodeCount=124, validCount=124 2024-10-16 13:42:14.882 INFO 4486 --- [metric-archive-async-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057260, nodeCount=141, validCount=141 2024-10-16 13:42:15.172 INFO 4486 --- [ocp-async-1,dcb109706ca7a5ac,a85385a77e598502] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:15.914 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:16.994 INFO 4486 --- [prometheus-minute-schedule-1,9daf6d22d7c14f18,d1a72affb4a2a675] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:17.001 INFO 4486 --- [prometheus-minute-schedule-1,9daf6d22d7c14f18,d1a72affb4a2a675] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:17.070 INFO 4486 --- [ocp-schedules-11,9d76125d560acf40,00d066c100d7bb36] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:42:17.143 INFO 4486 --- [ocp-schedules-4,7afa1a6ac3390da2,dfd0346f309cf4ff] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:42:17.173 INFO 4486 --- [ocp-async-5,a0f5c21db07ae9b1,393c6c5d0690a221] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:17.186 INFO 4486 --- [ocp-schedules-28,57a5b4d3aac41cd3,f2b2594d7a3171ed] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:42:17.469 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,7e55d64adf7203da,ba300cdb6380b39b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7e55d64adf7203da, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:17.470 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,7e55d64adf7203da,ba300cdb6380b39b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7e55d64adf7203da, duration=3 ms] 2024-10-16 13:42:18.915 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:18.981 INFO 4486 --- [ocp-schedules-13,6c2810e48d8b1b53,0f55bb8ec971c5b6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:42:19.173 INFO 4486 --- [ocp-async-10,b7d5f0bbf37c4a89,c72c535887b2db09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:19.618 INFO 4486 --- [ocp-schedules-24,d1fdf9f34a795f72,74f72e234d375d2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:42:19.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,51e1fda90ea0f4a3,177a1fd19c3d5cab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=51e1fda90ea0f4a3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:19.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,51e1fda90ea0f4a3,177a1fd19c3d5cab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=51e1fda90ea0f4a3, duration=1 ms] 2024-10-16 13:42:19.696 WARN 4486 --- [scanner-13271739-1,d1fdf9f34a795f72,8bb06eae79c2140e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:42:19.771 INFO 4486 --- [alarm-detect-13,d1fdf9f34a795f72,5b7b3d6f8818610c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057305 ~ 1729057329 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057320, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057320, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057320=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:42:19.994 INFO 4486 --- [prometheus-minute-schedule-1,992a77c851ba27e8,932ce87fbc9e0556] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:20.000 INFO 4486 --- [prometheus-minute-schedule-1,992a77c851ba27e8,932ce87fbc9e0556] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:20.609 INFO 4486 --- [ocp-schedules-26,485bb3df51ccc305,2fb56181a55b8102] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:42:20.811 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:42:20.869 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 13:42:20.869 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 13:42:20.996 INFO 4486 --- [prometheus-cache-monitor1,818a6b2ad48836f0,eda03ad47b02bfa9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:42:20.996 INFO 4486 --- [prometheus-cache-monitor1,818a6b2ad48836f0,eda03ad47b02bfa9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:42:21.173 INFO 4486 --- [ocp-async-2,c4492ef3729671ff,b69aae963eb7f20f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:21.208 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,085f8b7f721e2f22,df99d80e202e0054] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=085f8b7f721e2f22, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:21.208 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,085f8b7f721e2f22,df99d80e202e0054] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=085f8b7f721e2f22, duration=1 ms] 2024-10-16 13:42:21.561 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c03f4af02aa1a959,174887b24383f2be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c03f4af02aa1a959, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:21.562 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c03f4af02aa1a959,174887b24383f2be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c03f4af02aa1a959, duration=2 ms] 2024-10-16 13:42:21.916 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:22.994 INFO 4486 --- [prometheus-minute-schedule-1,87540f8dbb79278e,152623d3b1926914] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:23.001 INFO 4486 --- [ocp-async-2,ec72b147952ecbb9,b7634c3a07f73b7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:42:23.003 INFO 4486 --- [prometheus-minute-schedule-1,87540f8dbb79278e,152623d3b1926914] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:23.093 INFO 4486 --- [ocp-schedules-9,4b64e411f9bfd9d2,1cc6c70e2a0620c3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:42:23.173 INFO 4486 --- [ocp-async-9,f798b270f32ebd86,24ca4fec6993a429] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:24.530 INFO 4486 --- [ocp-schedules-23,de09e80739203014,66abd97bee437fd3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:42:24.917 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:25.173 INFO 4486 --- [ocp-async-4,97a31f46cb58b06a,2ac9c12542310606] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:25.227 INFO 4486 --- [ocp-schedules-18,f6b883d389b6bcb1,dd3764aa00945539] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:42:25.227 INFO 4486 --- [ocp-schedules-18,f6b883d389b6bcb1,dd3764aa00945539] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:42:25.261 INFO 4486 --- [Collect all tenants major compactions1,f6b883d389b6bcb1,7bfe760ec785de7d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:42:25.262 INFO 4486 --- [Collect all tenants major compactions2,f6b883d389b6bcb1,f419b2a70df792f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:42:25.262 INFO 4486 --- [Collect all tenants major compactions3,f6b883d389b6bcb1,9cd2cb36bccf7aa8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:42:25.263 INFO 4486 --- [Collect all tenants major compactions4,f6b883d389b6bcb1,4739b6bf87a68f19] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:42:25.264 INFO 4486 --- [Collect all tenants major compactions5,f6b883d389b6bcb1,93bc94b0c68d80c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:42:25.275 INFO 4486 --- [Collect all tenants major compactions1,f6b883d389b6bcb1,7bfe760ec785de7d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.276 INFO 4486 --- [Collect all tenants major compactions2,f6b883d389b6bcb1,f419b2a70df792f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.277 INFO 4486 --- [Collect all tenants major compactions2,f6b883d389b6bcb1,f419b2a70df792f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:42:25.277 INFO 4486 --- [Collect all tenants major compactions1,f6b883d389b6bcb1,7bfe760ec785de7d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:42:25.281 INFO 4486 --- [Collect all tenants major compactions2,f6b883d389b6bcb1,f419b2a70df792f9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:42:25.281 INFO 4486 --- [Collect all tenants major compactions1,f6b883d389b6bcb1,7bfe760ec785de7d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:42:25.281 INFO 4486 --- [Collect all tenants major compactions2,f6b883d389b6bcb1,f419b2a70df792f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:42:25.281 INFO 4486 --- [Collect all tenants major compactions1,f6b883d389b6bcb1,7bfe760ec785de7d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:42:25.282 INFO 4486 --- [Collect all tenants major compactions3,f6b883d389b6bcb1,9cd2cb36bccf7aa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.284 INFO 4486 --- [Collect all tenants major compactions3,f6b883d389b6bcb1,9cd2cb36bccf7aa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:42:25.288 INFO 4486 --- [Collect all tenants major compactions3,f6b883d389b6bcb1,9cd2cb36bccf7aa8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:42:25.288 INFO 4486 --- [Collect all tenants major compactions3,f6b883d389b6bcb1,9cd2cb36bccf7aa8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:42:25.296 INFO 4486 --- [Collect all tenants major compactions4,f6b883d389b6bcb1,4739b6bf87a68f19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.298 INFO 4486 --- [Collect all tenants major compactions4,f6b883d389b6bcb1,4739b6bf87a68f19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:42:25.298 INFO 4486 --- [Collect all tenants major compactions5,f6b883d389b6bcb1,93bc94b0c68d80c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.300 INFO 4486 --- [Collect all tenants major compactions5,f6b883d389b6bcb1,93bc94b0c68d80c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:42:25.302 INFO 4486 --- [Collect all tenants major compactions4,f6b883d389b6bcb1,4739b6bf87a68f19] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:42:25.302 INFO 4486 --- [Collect all tenants major compactions4,f6b883d389b6bcb1,4739b6bf87a68f19] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:42:25.305 INFO 4486 --- [Collect all tenants major compactions5,f6b883d389b6bcb1,93bc94b0c68d80c4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:42:25.305 INFO 4486 --- [Collect all tenants major compactions5,f6b883d389b6bcb1,93bc94b0c68d80c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:42:25.307 INFO 4486 --- [ocp-schedules-18,f6b883d389b6bcb1,dd3764aa00945539] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:42:25.462 INFO 4486 --- [ocp-schedules-28,055f8ae12d96cf60,790bbdd3950e6b21] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:42:25.463 INFO 4486 --- [ocp-schedules-28,055f8ae12d96cf60,790bbdd3950e6b21] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:25.499 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:42:25.499 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.503 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:42:25.511 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.512 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:42:25.516 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:42:25.516 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.521 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:42:25.522 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.523 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:42:25.531 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.532 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:42:25.534 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.535 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:42:25.540 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.541 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:42:25.545 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.546 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:42:25.549 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.550 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:25.558 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.559 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:25.560 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.561 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:25.565 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:42:25.566 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.570 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:42:25.571 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.571 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.571 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:42:25.572 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:25.582 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.582 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:42:25.583 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.583 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.584 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:42:25.584 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:42:25.598 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.598 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.599 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:42:25.602 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:42:25.611 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.612 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.612 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.613 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.613 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:42:25.613 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.614 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.615 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.616 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.616 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.617 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.617 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.618 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:42:25.618 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.618 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.619 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.620 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.620 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.621 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.622 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.622 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.623 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.624 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.624 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.625 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.626 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.626 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.627 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.627 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.628 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:42:25.628 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:42:25.629 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.630 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.634 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.635 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.636 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:42:25.638 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.638 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.639 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.639 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:42:25.641 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.642 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.642 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.642 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:25.645 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.646 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.648 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.649 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.651 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.652 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.652 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.653 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:25.653 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:25.653 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.654 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:42:25.654 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.655 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.658 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.659 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.661 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:42:25.662 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.663 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:25.663 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.665 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:42:25.669 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:42:25.669 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:42:25.693 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:25.695 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.695 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.696 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:42:25.696 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.700 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.701 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.704 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.705 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.708 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.709 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.711 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.712 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.712 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:42:25.712 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.712 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:42:25.712 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.712 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:42:25.712 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.714 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.715 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.715 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:42:25.715 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:42:25.718 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.719 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:42:25.722 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.723 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:42:25.726 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.727 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:42:25.728 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:42:25.731 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:42:25.792 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:25.794 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:25.795 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:42:25.804 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.804 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:42:25.804 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.804 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:42:25.804 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:42:25.804 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:42:25.808 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:42:25.808 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:42:25.869 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,54ce7e5c5c6d01bd,e64d78177288f1e7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=54ce7e5c5c6d01bd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:42:25.869 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,54ce7e5c5c6d01bd,e64d78177288f1e7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=54ce7e5c5c6d01bd, duration=2 ms] 2024-10-16 13:42:25.994 INFO 4486 --- [prometheus-minute-schedule-1,10e6a684bf4cabb5,4560b8cacbefa403] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:26.001 INFO 4486 --- [prometheus-minute-schedule-1,10e6a684bf4cabb5,4560b8cacbefa403] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:27.174 INFO 4486 --- [ocp-async-3,327e990284a07b4b,4cfa66b31607c94a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:27.202 INFO 4486 --- [ocp-schedules-22,b752bbbcba48e985,8697238f3802b8dd] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:42:27.202+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:42:27.205 INFO 4486 --- [ocp-schedules-22,b752bbbcba48e985,8697238f3802b8dd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:42:27.206 INFO 4486 --- [ocp-schedules-22,b752bbbcba48e985,8697238f3802b8dd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:42:27.207 INFO 4486 --- [ocp-schedules-22,b752bbbcba48e985,8697238f3802b8dd] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:42:27.478 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e7ce61cd867235d8,11997747d5c4574f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e7ce61cd867235d8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:27.479 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e7ce61cd867235d8,11997747d5c4574f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e7ce61cd867235d8, duration=3 ms] 2024-10-16 13:42:27.918 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:28.994 INFO 4486 --- [prometheus-minute-schedule-1,18de2f2e3a104fe4,27fbe02a7bd67637] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:29.001 INFO 4486 --- [prometheus-minute-schedule-1,18de2f2e3a104fe4,27fbe02a7bd67637] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:29.174 INFO 4486 --- [ocp-async-2,510ed839cda237b4,9b5fe744afdd30cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:29.637 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,015c437454ff6eef,a383bfd0b06862c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=015c437454ff6eef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:29.637 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,015c437454ff6eef,a383bfd0b06862c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=015c437454ff6eef, duration=1 ms] 2024-10-16 13:42:29.886 INFO 4486 --- [ocp-schedules-10,5f6679f914c9f677,d90157b0a881524b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:42:29.915 WARN 4486 --- [scanner-13272048-1,5f6679f914c9f677,4ed746bf33d2483d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:42:29.959 INFO 4486 --- [alarm-detect-19,5f6679f914c9f677,d2b3fa80e97c778b] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057320 ~ 1729057339 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057335, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057335, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057335=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:42:30.103 INFO 4486 --- [prometheus-second-schedule-1,5e1dbe4d25284208,802a5d9404f3ff89] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:42:30.615 INFO 4486 --- [ocp-schedules-4,6e912764c21e810c,8454e404869dda2c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:42:30.778 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,264d8a74e317ebf0,3e0fab26901eff5a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=264d8a74e317ebf0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:42:30.779 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,264d8a74e317ebf0,3e0fab26901eff5a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=264d8a74e317ebf0, duration=2 ms] 2024-10-16 13:42:30.920 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:31.174 INFO 4486 --- [ocp-async-9,f01354ecd552308d,17d03463a9d21134] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:31.213 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e34cc92ea7aea4dc,7373f3008eefef64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e34cc92ea7aea4dc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:31.213 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e34cc92ea7aea4dc,7373f3008eefef64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e34cc92ea7aea4dc, duration=1 ms] 2024-10-16 13:42:31.566 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,8c44430a338264e0,4204ec477a11ee83] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8c44430a338264e0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:31.566 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,8c44430a338264e0,4204ec477a11ee83] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8c44430a338264e0, duration=1 ms] 2024-10-16 13:42:31.994 INFO 4486 --- [prometheus-minute-schedule-1,75164ac48a90ba5b,845de21a551a63f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:32.003 INFO 4486 --- [prometheus-minute-schedule-1,75164ac48a90ba5b,845de21a551a63f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:33.001 INFO 4486 --- [ocp-async-8,3e62165520567a69,e4e17ae044ae60d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:42:33.174 INFO 4486 --- [ocp-async-10,ce0bee29e36a8586,a514d6d38a8c019a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:33.512 INFO 4486 --- [ocp-schedules-19,53ba7232b31fbbb1,0d057e2ca1f59efb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:42:33.514 INFO 4486 --- [ocp-schedules-19,53ba7232b31fbbb1,0d057e2ca1f59efb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:33.515 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:42:33.515 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:42:33.515 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:42:33.922 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:34.994 INFO 4486 --- [prometheus-minute-schedule-1,bc9e9d5e67461ab2,d2b55d87f20297f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:35.003 INFO 4486 --- [prometheus-minute-schedule-1,bc9e9d5e67461ab2,d2b55d87f20297f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:35.175 INFO 4486 --- [ocp-async-1,d11ad76f3b9ee98b,ed97a2a8530cba61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:36.923 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:37.063 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:42:37.064 INFO 4486 --- [ocp-schedules-26,34deedc60e0f1acc,a2ed3f78be429133] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:42:37.066 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:42:37.066 INFO 4486 --- [ocp-schedules-22,7175657ca0e025c7,77ff40f0bba81dde] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:42:37.068 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:42:37.068 INFO 4486 --- [ocp-schedules-22,7175657ca0e025c7,77ff40f0bba81dde] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:37.071 INFO 4486 --- [ocp-schedules-26,34deedc60e0f1acc,a2ed3f78be429133] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:42:37.081 INFO 4486 --- [ocp-schedules-15,8cac261be93f51ce,d0e79d00283baeb7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:42:37.083 INFO 4486 --- [ocp-schedules-15,8cac261be93f51ce,d0e79d00283baeb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:42:37.091 INFO 4486 --- [ocp-schedules-21,c7fa78049f5067a2,94611da99132760e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:42:37.093 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:42:37.093 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:42:37.093 INFO 4486 --- [ocp-schedules-11,4ea47f770a950c1c,21fa65663a90223f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:42:37.093 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:42:37.094 INFO 4486 --- [ocp-schedules-3,81312a487580327c,7cf2e55c64d241e7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:42:37.094 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:42:37.095 INFO 4486 --- [host-check-clockdiff1,81312a487580327c,2c192e6e3bf5aee1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:42:37.095 INFO 4486 --- [host-check-clockdiff2,81312a487580327c,d3c84d3a7b6a2c5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:42:37.096 INFO 4486 --- [host-check-agent-status1,4ea47f770a950c1c,6a48fb241d6f05a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:42:37.096 INFO 4486 --- [host-check-agent-status3,4ea47f770a950c1c,9f9afc774f0b1ade] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:42:37.099 INFO 4486 --- [host-check-agent-status2,4ea47f770a950c1c,f700a79d043afa53] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:42:37.099 INFO 4486 --- [host-check-clockdiff4,81312a487580327c,2e91c5caadfc499b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:42:37.099 INFO 4486 --- [host-check-agent-status4,4ea47f770a950c1c,c97c72613c019896] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:42:37.099 INFO 4486 --- [host-check-clockdiff3,81312a487580327c,c28549c1f0daece9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:42:37.104 INFO 4486 --- [host-check-agent-status1,4ea47f770a950c1c,6a48fb241d6f05a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:37.104 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:37.105 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:37.105 INFO 4486 --- [host-check-agent-status3,4ea47f770a950c1c,9f9afc774f0b1ade] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:37.106 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:37.106 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:37.128 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:37.128 INFO 4486 --- [host-check-agent-status1,4ea47f770a950c1c,6a48fb241d6f05a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:37.128 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:37.128 INFO 4486 --- [host-check-agent-status3,4ea47f770a950c1c,9f9afc774f0b1ade] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:37.128 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:37.126 INFO 4486 --- [host-check-clockdiff3,81312a487580327c,c28549c1f0daece9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057357 4 4 , error= 2024-10-16 13:42:37.132 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:37.132 INFO 4486 --- [host-check-agent-status4,4ea47f770a950c1c,c97c72613c019896] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:37.132 INFO 4486 --- [host-check-clockdiff1,81312a487580327c,2c192e6e3bf5aee1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057357 0 0 , error= 2024-10-16 13:42:37.132 INFO 4486 --- [host-check-clockdiff2,81312a487580327c,d3c84d3a7b6a2c5b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057357 4 4 , error= 2024-10-16 13:42:37.137 INFO 4486 --- [host-check-agent-status2,4ea47f770a950c1c,f700a79d043afa53] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:37.137 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.138 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:37.139 INFO 4486 --- [host-check-agent-status2,4ea47f770a950c1c,f700a79d043afa53] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:37.139 INFO 4486 --- [host-check-agent-status4,4ea47f770a950c1c,c97c72613c019896] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:37.140 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:42:37.146 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:37.209 INFO 4486 --- [ocp-async-4,7118b52df8521d1f,eec67f53f5546ef2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:37.218 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:42:37.218 INFO 4486 --- [host-check-agent-status3,4ea47f770a950c1c,9f9afc774f0b1ade] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:42:37.219 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:37.220 INFO 4486 --- [host-check-agent-status4,4ea47f770a950c1c,c97c72613c019896] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:42:37.221 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.221 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:42:37.221 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:42:37.221242735+08:00 2024-10-16 13:42:37.222 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.222 INFO 4486 --- [host-check-agent-status2,4ea47f770a950c1c,f700a79d043afa53] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:42:37.222 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:42:37.222 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:42:37.222+08:00 2024-10-16 13:42:37.222 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:42:37.223 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:37.223 INFO 4486 --- [host-check-agent-status1,4ea47f770a950c1c,6a48fb241d6f05a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:42:37.224 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:42:37.227385850+08:00 2024-10-16 13:42:37.224 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:42:37.227207707+08:00 2024-10-16 13:42:37.224 INFO 4486 --- [host-check-agent-status3,4ea47f770a950c1c,9f9afc774f0b1ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:42:37.224 INFO 4486 --- [host-check-agent-status3,4ea47f770a950c1c,9f9afc774f0b1ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:42:37.224 INFO 4486 --- [host-check-agent-status3,4ea47f770a950c1c,9f9afc774f0b1ade] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:42:37.224 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:42:37.224+08:00 2024-10-16 13:42:37.224 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:42:37.224+08:00 2024-10-16 13:42:37.225 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:42:37.228047587+08:00 2024-10-16 13:42:37.225 INFO 4486 --- [host-check-agent-status4,4ea47f770a950c1c,c97c72613c019896] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:42:37.225 INFO 4486 --- [host-check-agent-status4,4ea47f770a950c1c,c97c72613c019896] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:42:37.225 INFO 4486 --- [host-check-agent-status4,4ea47f770a950c1c,c97c72613c019896] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:42:37.225 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:42:37.222+08:00 and status AVAILABLE 2024-10-16 13:42:37.225 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:42:37.225 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:42:37.225+08:00 2024-10-16 13:42:37.227 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:42:37.224+08:00 and status AVAILABLE 2024-10-16 13:42:37.227 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:42:37.224+08:00 and status AVAILABLE 2024-10-16 13:42:37.227 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:42:37.228 INFO 4486 --- [host-check-agent-status2,4ea47f770a950c1c,f700a79d043afa53] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:42:37.228 INFO 4486 --- [host-check-agent-status2,4ea47f770a950c1c,f700a79d043afa53] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:42:37.228 INFO 4486 --- [host-check-agent-status2,4ea47f770a950c1c,f700a79d043afa53] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:42:37.228 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:42:37.225+08:00 and status AVAILABLE 2024-10-16 13:42:37.228 INFO 4486 --- [host-check-agent-status1,4ea47f770a950c1c,6a48fb241d6f05a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:42:37.228 INFO 4486 --- [host-check-agent-status1,4ea47f770a950c1c,6a48fb241d6f05a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:42:37.228 INFO 4486 --- [host-check-agent-status1,4ea47f770a950c1c,6a48fb241d6f05a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:42:37.229 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:42:37.229 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:42:37.230 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:42:37.231 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:42:37.234 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.235 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:37.235 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.235 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:37.236 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:42:37.236 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:37.236 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:37.237 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:37.238 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:37.238 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.238 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.239 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:37.240 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:37.241 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:42:37.242 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:37.245 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.246 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:37.255 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.257 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:42:37.258 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:37.259 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.260 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.260 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.260 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.260 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.260 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:42:37.260 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.260 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.261 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:37.261 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:37.267 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:42:37.276 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.276 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:42:37.277 INFO 4486 --- [ocp-schedules-25,f6b0916c61e911b2,de6d9b2167d6400e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:42:37.287 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:42:37.298 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:42:37.301 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:42:37.311 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:42:37.312 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:42:37.314 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:37.315 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:42:37.323 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:42:37.325 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:42:37.328 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:37.340 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:37.342 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:42:37.343 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:42:37.352 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:42:37.365 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:42:37.369 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:37.378 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:42:37.382 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:42:37.393 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:42:37.399 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:37.402 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:42:37.403 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:42:37.409 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:37.413 INFO 4486 --- [host-check-host-status3,c7fa78049f5067a2,2e577b8660df935a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:42:37.415 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:42:37.432 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:42:37.433 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:42:37.445 INFO 4486 --- [host-check-host-status4,c7fa78049f5067a2,ca77c9ba0ba7ce7c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:42:37.450 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:42:37.461 INFO 4486 --- [host-check-host-status2,c7fa78049f5067a2,83396edef1894671] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:42:37.486 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:42:37.487 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1ae44401c4e4d7e7,324725ac8f1a7bcc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1ae44401c4e4d7e7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:37.488 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1ae44401c4e4d7e7,324725ac8f1a7bcc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1ae44401c4e4d7e7, duration=2 ms] 2024-10-16 13:42:37.498 INFO 4486 --- [host-check-host-status1,c7fa78049f5067a2,d9d28f05336464c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:42:37.501 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:37.503 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.504 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:42:37.505 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.506 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.507 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:42:37.518 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.518 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.518 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.518 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.518 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.518 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.545 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:42:37.585 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:42:37.630 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:37.645 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:37.646 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.647 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:42:37.648 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:42:37.649 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.650 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:42:37.651 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.651 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.652 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:42:37.664 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.664 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.664 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.664 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.664 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.665 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.673 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:42:37.842 INFO 4486 --- [ocp-schedules-30,693650cd5b5f142c,5dbe9085af76d5cd] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-10-16 13:42:37.842 INFO 4486 --- [ocp-schedules-30,693650cd5b5f142c,5dbe9085af76d5cd] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-10-16 13:42:37.875 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:42:37.921 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:37.923 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.924 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:42:37.925 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.926 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.926 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:42:37.936 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.936 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.936 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.936 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.936 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:37.937 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:37.954 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:42:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:42:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:42:37.994 INFO 4486 --- [prometheus-minute-schedule-1,95e3f1a00df7e7d8,c35a2a62cce35469] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:38.002 INFO 4486 --- [prometheus-minute-schedule-1,95e3f1a00df7e7d8,c35a2a62cce35469] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:38.061 INFO 4486 --- [host-check-clockdiff4,81312a487580327c,2e91c5caadfc499b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057358 3 3 , error= 2024-10-16 13:42:38.167 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:42:38.212 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:38.215 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:38.216 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:42:38.216 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:38.217 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:38.218 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:42:38.228 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:38.228 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:38.228 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:38.228 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:38.228 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:42:38.229 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:38.293 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:42:38.494 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:42:38.297064, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:42:38.495 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:42:38.297064, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:42:38.533 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:42:38.297064} } 2024-10-16 13:42:38.596 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:42:38.603 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:42:38.606 INFO 4486 --- [ocp-schedules-29,a0128164fe9f3595,93beea8134d30c17] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:42:38.986 INFO 4486 --- [ocp-schedules-12,1cfe518091fccdc2,ff9641ae22438f8c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:42:39.209 INFO 4486 --- [ocp-async-10,0999b16733d72324,64b5e3cafe1cdda5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:39.657 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c774bcd760f40aaf,5a7069774b0161b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c774bcd760f40aaf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:39.657 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c774bcd760f40aaf,5a7069774b0161b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c774bcd760f40aaf, duration=2 ms] 2024-10-16 13:42:39.924 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:40.006 INFO 4486 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:42:40.006 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:42:40.018 INFO 4486 --- [ocp-schedules-4,c7a01628545e12e7,5ba3667bd7566926] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:42:40.028 INFO 4486 --- [Check obproxy1,33541189e943c5cd,f6ae67d5b38285d8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:42:40.028 INFO 4486 --- [Check obproxy2,051f85a045b08f50,dcaf5c5c37016331] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:42:40.029 INFO 4486 --- [Check obproxy1,33541189e943c5cd,f6ae67d5b38285d8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:40.029 INFO 4486 --- [Check obproxy2,051f85a045b08f50,dcaf5c5c37016331] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:40.031 INFO 4486 --- [Collect obproxy1,eb09a7d8918c29cf,8709947638d8612f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:42:40.031 INFO 4486 --- [Collect obproxy1,eb09a7d8918c29cf,8709947638d8612f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:40.033 INFO 4486 --- [Collect obproxy2,abe7c2ff7c0d9905,0626ef8d51c27c23] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:42:40.033 INFO 4486 --- [Collect obproxy2,abe7c2ff7c0d9905,0626ef8d51c27c23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:42:40.056 INFO 4486 --- [Check obproxy1,33541189e943c5cd,f6ae67d5b38285d8] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:42:40.056 INFO 4486 --- [Check obproxy1,33541189e943c5cd,f6ae67d5b38285d8] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:42:40.056 INFO 4486 --- [Collect obproxy2,abe7c2ff7c0d9905,0626ef8d51c27c23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:42:40.059 INFO 4486 --- [Check obproxy2,051f85a045b08f50,dcaf5c5c37016331] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:42:40.059 INFO 4486 --- [Check obproxy2,051f85a045b08f50,dcaf5c5c37016331] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:42:40.060 INFO 4486 --- [Collect obproxy1,eb09a7d8918c29cf,8709947638d8612f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:42:40.061 INFO 4486 --- [Collect obproxy2,abe7c2ff7c0d9905,0626ef8d51c27c23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:42:40.063 INFO 4486 --- [Collect obproxy1,eb09a7d8918c29cf,8709947638d8612f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:42:40.066 INFO 4486 --- [Collect obproxy1,eb09a7d8918c29cf,8709947638d8612f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:42:40.067 INFO 4486 --- [Collect obproxy2,abe7c2ff7c0d9905,0626ef8d51c27c23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:42:40.070 WARN 4486 --- [scanner-13272318-1,c7a01628545e12e7,c8e4ea86a56b5b2e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:42:40.128 INFO 4486 --- [alarm-detect-10,c7a01628545e12e7,59541b292ffeb7c0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057335 ~ 1729057350 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057350, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057350, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057350=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:42:40.622 INFO 4486 --- [ocp-schedules-27,de2771ee1a1b9920,be39316936b27895] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:42:40.994 INFO 4486 --- [prometheus-minute-schedule-1,3f448ab78d1aaf5b,f578fbed4a4dc8c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:40.995 INFO 4486 --- [prometheus-hour-schedule-1,18ec590963047c80,1d6bc503c20fbef0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:42:41.001 INFO 4486 --- [prometheus-minute-schedule-1,3f448ab78d1aaf5b,f578fbed4a4dc8c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:41.209 INFO 4486 --- [ocp-async-1,399cd112f1ca9c96,6e2384690896978e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:41.219 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,eda23ab6707f80e9,e715f2ccf1a57562] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eda23ab6707f80e9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:41.219 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,eda23ab6707f80e9,e715f2ccf1a57562] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eda23ab6707f80e9, duration=1 ms] 2024-10-16 13:42:41.570 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b6e8117d5b3ab7f7,11ee0c68081a0071] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b6e8117d5b3ab7f7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:41.570 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b6e8117d5b3ab7f7,11ee0c68081a0071] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b6e8117d5b3ab7f7, duration=1 ms] 2024-10-16 13:42:42.114 INFO 4486 --- [ocp-schedules-14,13735894edd2c431,6e88ce60888b7161] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:42:42.925 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:43.001 INFO 4486 --- [ocp-async-5,b301250aab051a7f,09f307341d21150f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:42:43.210 INFO 4486 --- [ocp-async-8,f1486fc09a1f2650,b9dfb44aa5809c60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:43.244 INFO 4486 --- [ocp-schedules-16,72bfabf8b7f6612a,b5684414948242a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:42:43.245 INFO 4486 --- [ocp-schedules-16,3234c9a3a8f4fa88,dad549622caa66ea] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:42:43.246 INFO 4486 --- [ocp-schedules-16,3234c9a3a8f4fa88,dad549622caa66ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:42:43.271 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.271 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:43.281 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.282 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:43.307 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.307 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.308 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:42:43.310 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.312 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:43.337 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.338 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.338 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:43.339 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:42:43.339 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.342 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:42:43.344 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:43.400 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.400 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:43.412 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.421 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.422 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.422 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:43.424 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:42:43.428 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:42:43.433 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:43.435 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:43.462 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:42:43.480 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.482 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.483 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:42:43.504 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:42:43.504 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:42:43.516 INFO 4486 --- [ocp-schedules-19,53ba7232b31fbbb1,0d057e2ca1f59efb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:42:43.532 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.534 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.535 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:42:43.538 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.542 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:42:43.542 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:42:43.545 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:43.552 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.553 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.554 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:42:43.564 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:42:43.564 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:42:43.575 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.575 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:43.612 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.612 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:43.639 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.639 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:43.666 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.666 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:42:43.696 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.696 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:42:43.724 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:42:43.752 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.754 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.755 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:42:43.758 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:42:43.763 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:42:43.768 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:42:43.770 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:42:43.790 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:42:43.936 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:42:43.966 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:42:43.968 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:42:43.969 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:42:43.994 INFO 4486 --- [prometheus-minute-schedule-1,52e895819408367c,96c3561d729a25d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:44.001 INFO 4486 --- [prometheus-minute-schedule-1,52e895819408367c,96c3561d729a25d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:44.002 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:42:44.002 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:42:44.518 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,adb487e8459454dc,6d53e0c25734db30] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=adb487e8459454dc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:42:44.518 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,adb487e8459454dc,6d53e0c25734db30] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=adb487e8459454dc, duration=1 ms] 2024-10-16 13:42:45.210 INFO 4486 --- [ocp-async-7,e4bdea03814a1c46,d79b302e229d373b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:45.493 INFO 4486 --- [ocp-schedules-12,864a991be683e4a2,8acd6f5cb28cdd75] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:42:45.498 INFO 4486 --- [ocp-schedules-12,864a991be683e4a2,8acd6f5cb28cdd75] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:42:45.498 INFO 4486 --- [ocp-schedules-12,864a991be683e4a2,8acd6f5cb28cdd75] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:42:45.514 INFO 4486 --- [ocp-schedules-11,81741f0734fb4be4,28f951e1b03b9288] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:42:45.514 INFO 4486 --- [ocp-schedules-11,81741f0734fb4be4,28f951e1b03b9288] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:42:45.927 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:46.994 INFO 4486 --- [prometheus-minute-schedule-1,cd4d585d7d73c314,68a75b0067b8f5f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:47.000 INFO 4486 --- [prometheus-minute-schedule-1,cd4d585d7d73c314,68a75b0067b8f5f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:47.070 INFO 4486 --- [ocp-schedules-22,7175657ca0e025c7,77ff40f0bba81dde] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:42:47.147 INFO 4486 --- [ocp-schedules-9,d5517d46cb38ae26,58acba9a9396a00b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:42:47.210 INFO 4486 --- [ocp-async-2,cedf9fb237043c14,cc5fbab8291fcf32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:47.229 INFO 4486 --- [ocp-schedules-15,8cac261be93f51ce,d0e79d00283baeb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:42:47.492 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,13da068dca9b3544,34ac1059f762f40a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=13da068dca9b3544, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:47.493 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,13da068dca9b3544,34ac1059f762f40a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=13da068dca9b3544, duration=2 ms] 2024-10-16 13:42:48.928 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:49.210 INFO 4486 --- [ocp-async-8,06f6853319be8ea5,74e0bb06e9cca20d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:49.667 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,93c9bd272cb35152,4eac5abcf58d648f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=93c9bd272cb35152, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:49.667 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,93c9bd272cb35152,4eac5abcf58d648f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=93c9bd272cb35152, duration=3 ms] 2024-10-16 13:42:49.994 INFO 4486 --- [prometheus-minute-schedule-1,ddaa5a9def1205d3,d3bf2a2dbc97c795] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:50.002 INFO 4486 --- [prometheus-minute-schedule-1,ddaa5a9def1205d3,d3bf2a2dbc97c795] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:50.231 INFO 4486 --- [ocp-schedules-6,ba5b13ca2a981af9,debfcdf5f2e0d2e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:42:50.280 WARN 4486 --- [scanner-13272643-1,ba5b13ca2a981af9,9f30a5b657db0f5e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:42:50.437 INFO 4486 --- [ocp-schedules-11,dbd4cad6f02ddc85,4384de4982ab3ca0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:42:50.437 INFO 4486 --- [ocp-schedules-11,dbd4cad6f02ddc85,4384de4982ab3ca0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:42:50.438 INFO 4486 --- [ocp-schedules-11,dbd4cad6f02ddc85,4384de4982ab3ca0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:42:50.438 INFO 4486 --- [ocp-schedules-11,dbd4cad6f02ddc85,4384de4982ab3ca0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:42:50.439 INFO 4486 --- [ocp-schedules-11,dbd4cad6f02ddc85,4384de4982ab3ca0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:42:50.439 INFO 4486 --- [ocp-schedules-11,dbd4cad6f02ddc85,4384de4982ab3ca0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:42:50.469 INFO 4486 --- [ocp-schedules-29,77c765df79c9c705,ea4e2f6073295f55] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:42:50.469 INFO 4486 --- [ocp-schedules-29,77c765df79c9c705,ea4e2f6073295f55] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:42:50.469 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:42:50.469 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:42:50.469 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:42:50.469 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:42:50.469 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:42:50.475 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:42:50.475 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:50.475 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:50.476 INFO 4486 --- [ocp-schedules-29,77c765df79c9c705,ea4e2f6073295f55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:42:50.476 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:50.477 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:50.477 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:42:50.477 INFO 4486 --- [ocp-schedules-29,77c765df79c9c705,ea4e2f6073295f55] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:42:50.477 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:50.477 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:50.478 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:50.479 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:50.503 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:42:50.503 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:42:50.504 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:42:50.504 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:42:50.504 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:42:50.504 INFO 4486 --- [ocp-schedules-29,77c765df79c9c705,ea4e2f6073295f55] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:42:50.509 INFO 4486 --- [ocp-schedules-29,77c765df79c9c705,ea4e2f6073295f55] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:42:50.628 INFO 4486 --- [ocp-schedules-3,110254c543f9032d,70f163c0c2f4957d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:42:50.651 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:42:50.651 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:42:50.666 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:42:50.667 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:42:50.667 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:42:50.679 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:42:50.680 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:42:50.680 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:42:50.684 INFO 4486 --- [ocp-schedules-10,3ff5eaff931b66e1,73b8e809a3bc8030] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:42:51.211 INFO 4486 --- [ocp-async-3,6d866ebd45b8d58e,4e68440d8024d8ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:51.228 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d2bfed4aa4ca876a,c2a48f561b9ed3d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d2bfed4aa4ca876a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:51.228 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d2bfed4aa4ca876a,c2a48f561b9ed3d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d2bfed4aa4ca876a, duration=1 ms] 2024-10-16 13:42:51.574 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6ab9a77e523fa317,76292561c1e99f8d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6ab9a77e523fa317, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:51.574 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6ab9a77e523fa317,76292561c1e99f8d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6ab9a77e523fa317, duration=1 ms] 2024-10-16 13:42:51.962 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:52.994 INFO 4486 --- [prometheus-minute-schedule-1,7877ab56f2114821,e62b4cd897f2c244] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:53.000 INFO 4486 --- [prometheus-minute-schedule-1,7877ab56f2114821,e62b4cd897f2c244] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:53.001 INFO 4486 --- [ocp-async-3,0a62534b6f746220,39f279721de62363] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:42:53.211 INFO 4486 --- [ocp-async-1,eb91aad82b811b33,4ee3043db41343fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:54.963 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:55.211 INFO 4486 --- [ocp-async-8,4424ec11cda5e327,2d629e50edc76b4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:55.994 INFO 4486 --- [prometheus-minute-schedule-1,a80477504e565bef,abcb71c56b688b04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:56.002 INFO 4486 --- [prometheus-minute-schedule-1,a80477504e565bef,abcb71c56b688b04] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:57.211 INFO 4486 --- [ocp-async-7,ef087518e34a6a4d,cf5c455e53276a45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:57.498 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f56dbbe1175984a2,1d972b7008bf1277] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f56dbbe1175984a2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:57.498 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f56dbbe1175984a2,1d972b7008bf1277] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f56dbbe1175984a2, duration=2 ms] 2024-10-16 13:42:57.965 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:42:58.992 INFO 4486 --- [ocp-schedules-6,957ba989b61506f2,eef7097a9279aac0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:42:58.994 INFO 4486 --- [prometheus-minute-schedule-1,8a5723d4b27dd3b1,d6a62400cadb29f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057380, interval=60 2024-10-16 13:42:59.002 INFO 4486 --- [prometheus-minute-schedule-1,8a5723d4b27dd3b1,d6a62400cadb29f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:42:59.212 INFO 4486 --- [ocp-async-5,6b6923aad44a6d26,bc02eb6ed51079de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:42:59.449 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:42:59.466 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:42:59.468 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:42:59.495 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:42:59.497 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:42:59.522 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:42:59.524 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:42:59.554 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:42:59.556 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:42:59.587 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:42:59.589 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:42:59.611 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:42:59.613 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:42:59.641 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:42:59.642 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:42:59.668 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:42:59.670 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:42:59.672 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,14b3192f1e6b9f99,aa29cfe52704e12d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=14b3192f1e6b9f99, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:42:59.673 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,14b3192f1e6b9f99,aa29cfe52704e12d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=14b3192f1e6b9f99, duration=1 ms] 2024-10-16 13:42:59.699 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:42:59.702 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:42:59.734 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:42:59.736 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:42:59.758 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:42:59.759 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:42:59.783 INFO 4486 --- [ocp-schedules-22,3b4acc37ddfd9f9e,4b90b33ca2b5ae35] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:43:00.372 INFO 4486 --- [ocp-schedules-30,6ffa19779d66969b,51c1ced811fef141] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:43:00.418 WARN 4486 --- [scanner-13272886-1,6ffa19779d66969b,3ca05156eff687c9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:43:00.503 INFO 4486 --- [alarm-detect-15,6ffa19779d66969b,92dc93f5752b2d03] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057350 ~ 1729057370 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057365, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057365, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057365=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:43:00.636 INFO 4486 --- [ocp-schedules-18,e33b0b012374800c,69423e418ea9053f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:43:00.966 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:01.212 INFO 4486 --- [ocp-async-9,1596c411fe23aeb4,52bf14798753d2a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:01.233 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b19efba81b8bb143,6cbbea08c13fa1c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b19efba81b8bb143, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:01.233 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b19efba81b8bb143,6cbbea08c13fa1c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b19efba81b8bb143, duration=1 ms] 2024-10-16 13:43:01.578 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ba56f0097009ea8d,7507bc9fb5b11fc4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ba56f0097009ea8d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:01.578 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ba56f0097009ea8d,7507bc9fb5b11fc4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ba56f0097009ea8d, duration=1 ms] 2024-10-16 13:43:01.630 INFO 4486 --- [ocp-schedules-23,9305000b2abbb1a1,e62a1c53a43d50e3] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:43:01.630+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:43:01.634 INFO 4486 --- [ocp-schedules-23,9305000b2abbb1a1,e62a1c53a43d50e3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:43:01.644 INFO 4486 --- [ocp-schedules-23,9305000b2abbb1a1,e62a1c53a43d50e3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:43:01.655 INFO 4486 --- [ocp-schedules-23,9305000b2abbb1a1,e62a1c53a43d50e3] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:43:01.994 INFO 4486 --- [prometheus-minute-schedule-1,c2e6ed11b81b62f2,0fcfa42b6a3815e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:02.000 INFO 4486 --- [prometheus-minute-schedule-1,c2e6ed11b81b62f2,0fcfa42b6a3815e5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:03.002 INFO 4486 --- [ocp-async-6,73c0005c38c25f38,552039207dddb1a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:43:03.212 INFO 4486 --- [ocp-async-3,b050e3b0d9cf8f5d,e0cea4fc971d7f13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:03.967 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:04.994 INFO 4486 --- [prometheus-minute-schedule-1,6b2a7562446a0772,73aad03a6754f50f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:05.001 INFO 4486 --- [prometheus-minute-schedule-1,6b2a7562446a0772,73aad03a6754f50f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:05.212 INFO 4486 --- [ocp-async-10,887e201e70054569,d398f426b29c58a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:06.969 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:07.066 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:43:07.067 INFO 4486 --- [ocp-schedules-9,9bb1d9885d4c33f5,71de8e8d40302b11] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:43:07.069 INFO 4486 --- [ocp-schedules-9,9bb1d9885d4c33f5,71de8e8d40302b11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:07.081 INFO 4486 --- [ocp-schedules-18,36436759d8eeccbc,0c217c33f189a594] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:43:07.083 INFO 4486 --- [ocp-schedules-18,36436759d8eeccbc,0c217c33f189a594] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:43:07.089 INFO 4486 --- [ocp-schedules-27,5379049fb2bdbdb0,e4cf3a806328cd44] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:43:07.090 INFO 4486 --- [ocp-schedules-7,b0906a04ed59c005,a08b92242568019f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:43:07.090 INFO 4486 --- [ocp-schedules-30,e98ecfa90012a928,51108e93e7669c55] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:43:07.090 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:43:07.090 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:43:07.090 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:43:07.091 INFO 4486 --- [host-check-clockdiff1,e98ecfa90012a928,ae38a7c0886e4907] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:43:07.091 INFO 4486 --- [host-check-agent-status1,b0906a04ed59c005,78abd8a8cfd05f47] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:43:07.091 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:43:07.091 INFO 4486 --- [host-check-clockdiff2,e98ecfa90012a928,123f0ceea37c1b39] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:43:07.091 INFO 4486 --- [host-check-agent-status2,b0906a04ed59c005,2a414ab420ab4e87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:43:07.092 INFO 4486 --- [host-check-agent-status3,b0906a04ed59c005,347dae59486d6eee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:43:07.092 INFO 4486 --- [host-check-agent-status4,b0906a04ed59c005,de9a4d3f5dbda130] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:43:07.092 INFO 4486 --- [host-check-clockdiff3,e98ecfa90012a928,fdfa642a952caa22] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:43:07.094 INFO 4486 --- [host-check-clockdiff4,e98ecfa90012a928,2b18d14030edbc03] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:43:07.098 INFO 4486 --- [host-check-agent-status1,b0906a04ed59c005,78abd8a8cfd05f47] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:07.098 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:07.098 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:07.098 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:43:07.098 INFO 4486 --- [host-check-agent-status3,b0906a04ed59c005,347dae59486d6eee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:43:07.099 INFO 4486 --- [host-check-agent-status4,b0906a04ed59c005,de9a4d3f5dbda130] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:07.099 INFO 4486 --- [host-check-agent-status2,b0906a04ed59c005,2a414ab420ab4e87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:43:07.105 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:43:07.121 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:43:07.121 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:07.121 INFO 4486 --- [host-check-agent-status1,b0906a04ed59c005,78abd8a8cfd05f47] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:07.136 INFO 4486 --- [host-check-agent-status2,b0906a04ed59c005,2a414ab420ab4e87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:43:07.136 INFO 4486 --- [host-check-agent-status4,b0906a04ed59c005,de9a4d3f5dbda130] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:07.136 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:07.139 INFO 4486 --- [host-check-agent-status3,b0906a04ed59c005,347dae59486d6eee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:43:07.139 INFO 4486 --- [host-check-clockdiff3,e98ecfa90012a928,fdfa642a952caa22] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057387 4 4 , error= 2024-10-16 13:43:07.139 INFO 4486 --- [host-check-clockdiff4,e98ecfa90012a928,2b18d14030edbc03] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057387 3 3 , error= 2024-10-16 13:43:07.139 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:43:07.142 INFO 4486 --- [host-check-clockdiff1,e98ecfa90012a928,ae38a7c0886e4907] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057387 0 0 , error= 2024-10-16 13:43:07.142 INFO 4486 --- [host-check-clockdiff2,e98ecfa90012a928,123f0ceea37c1b39] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057387 4 4 , error= 2024-10-16 13:43:07.146 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.147 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:07.148 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:43:07.164 INFO 4486 --- [host-check-agent-status1,b0906a04ed59c005,78abd8a8cfd05f47] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:43:07.165 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.165 INFO 4486 --- [host-check-agent-status2,b0906a04ed59c005,2a414ab420ab4e87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:43:07.166 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:43:07.166 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:43:07.166 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:07.166 INFO 4486 --- [host-check-agent-status4,b0906a04ed59c005,de9a4d3f5dbda130] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:43:07.169 INFO 4486 --- [host-check-agent-status3,b0906a04ed59c005,347dae59486d6eee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:43:07.170 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:43:07.170 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:43:07.171 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:43:07.170149245+08:00 2024-10-16 13:43:07.171 INFO 4486 --- [host-check-agent-status1,b0906a04ed59c005,78abd8a8cfd05f47] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:43:07.171 INFO 4486 --- [host-check-agent-status1,b0906a04ed59c005,78abd8a8cfd05f47] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:43:07.171 INFO 4486 --- [host-check-agent-status1,b0906a04ed59c005,78abd8a8cfd05f47] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:43:07.171 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:43:07.173826281+08:00 2024-10-16 13:43:07.172 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.172 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:43:07.171+08:00 2024-10-16 13:43:07.172 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:43:07.171+08:00 2024-10-16 13:43:07.173 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:43:07.176195257+08:00 2024-10-16 13:43:07.173 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status4,b0906a04ed59c005,de9a4d3f5dbda130] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status2,b0906a04ed59c005,2a414ab420ab4e87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status2,b0906a04ed59c005,2a414ab420ab4e87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status4,b0906a04ed59c005,de9a4d3f5dbda130] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status2,b0906a04ed59c005,2a414ab420ab4e87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status4,b0906a04ed59c005,de9a4d3f5dbda130] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:43:07.173+08:00 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status3,b0906a04ed59c005,347dae59486d6eee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status3,b0906a04ed59c005,347dae59486d6eee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:43:07.176 INFO 4486 --- [host-check-agent-status3,b0906a04ed59c005,347dae59486d6eee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:43:07.177 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:43:07.171+08:00 and status AVAILABLE 2024-10-16 13:43:07.177 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:43:07.171+08:00 and status AVAILABLE 2024-10-16 13:43:07.178 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:43:07.181417137+08:00 2024-10-16 13:43:07.178 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:43:07.178+08:00 2024-10-16 13:43:07.179 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:43:07.180 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:43:07.173+08:00 and status AVAILABLE 2024-10-16 13:43:07.180 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:43:07.180 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:43:07.181 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:43:07.178+08:00 and status AVAILABLE 2024-10-16 13:43:07.182 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:43:07.183 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:43:07.187 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.188 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:07.188 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:43:07.189 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:07.190 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:07.190 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:43:07.191 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:43:07.191 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:07.191 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:43:07.192 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:43:07.198 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.199 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:43:07.208 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.209 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:43:07.210 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:07.211 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.212 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:07.212 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:43:07.212 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:07.212 INFO 4486 --- [ocp-async-1,7b04cb293a04c97b,c91d1be25ef9dbc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:07.213 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:07.225 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:07.226 INFO 4486 --- [ocp-schedules-21,dca661910a11f5bd,5c3ce8bae8e53f1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:43:07.246 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:43:07.247 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:43:07.254 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:43:07.257 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:43:07.257 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:43:07.258 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2780fbae4f026efc,e34b114233238f3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=2780fbae4f026efc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:43:07.259 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2780fbae4f026efc,e34b114233238f3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=2780fbae4f026efc, duration=3 ms] 2024-10-16 13:43:07.266 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:43:07.272 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:07.273 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:07.273 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:43:07.283 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:07.283 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:43:07.300 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:07.304 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:43:07.309 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:43:07.313 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:43:07.317 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:43:07.320 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:43:07.328 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:43:07.329 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:07.337 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:07.347 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:07.355 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:43:07.359 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:43:07.369 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:43:07.369 INFO 4486 --- [host-check-host-status3,5379049fb2bdbdb0,14999924612beac9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:43:07.371 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:43:07.382 INFO 4486 --- [host-check-host-status4,5379049fb2bdbdb0,fb4a0b7813f6fae8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:43:07.383 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:43:07.385 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:07.393 INFO 4486 --- [host-check-host-status2,5379049fb2bdbdb0,c17a07c25dc589d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:43:07.439 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:43:07.451 INFO 4486 --- [host-check-host-status1,5379049fb2bdbdb0,a47678929087b685] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:43:07.502 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f2a50c6bc47ee57e,7dbd7bd1fd7f61a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f2a50c6bc47ee57e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:07.502 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f2a50c6bc47ee57e,7dbd7bd1fd7f61a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f2a50c6bc47ee57e, duration=1 ms] 2024-10-16 13:43:07.994 INFO 4486 --- [prometheus-minute-schedule-1,8705bc0ceefdcfb9,08decedce4ba3b09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:08.001 INFO 4486 --- [prometheus-minute-schedule-1,8705bc0ceefdcfb9,08decedce4ba3b09] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:09.213 INFO 4486 --- [ocp-async-8,45a40b3e5975fe4f,8352c8162483c1c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:09.683 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,29c3171d859e87e1,fde8ac327e935ab1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=29c3171d859e87e1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:09.684 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,29c3171d859e87e1,fde8ac327e935ab1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=29c3171d859e87e1, duration=2 ms] 2024-10-16 13:43:09.970 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:10.005 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:43:10.005 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:43:10.024 INFO 4486 --- [Check obproxy2,96ed9a2e9485eefe,2448e99f3ca8ec6a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:43:10.025 INFO 4486 --- [Check obproxy2,96ed9a2e9485eefe,2448e99f3ca8ec6a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:10.026 INFO 4486 --- [Check obproxy1,cff48d57468fbd4f,4098c589aed8968b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:43:10.026 INFO 4486 --- [Check obproxy1,cff48d57468fbd4f,4098c589aed8968b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:10.032 INFO 4486 --- [Collect obproxy2,a5075d53d10aad60,edccf94e97d54d6e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:43:10.032 INFO 4486 --- [Collect obproxy2,a5075d53d10aad60,edccf94e97d54d6e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:10.032 INFO 4486 --- [Collect obproxy1,d76c8aa4291c174a,2e6d04e33b8a632d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:43:10.032 INFO 4486 --- [Collect obproxy1,d76c8aa4291c174a,2e6d04e33b8a632d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:10.049 INFO 4486 --- [Check obproxy2,96ed9a2e9485eefe,2448e99f3ca8ec6a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:43:10.049 INFO 4486 --- [Check obproxy2,96ed9a2e9485eefe,2448e99f3ca8ec6a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:43:10.056 INFO 4486 --- [Collect obproxy2,a5075d53d10aad60,edccf94e97d54d6e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:43:10.061 INFO 4486 --- [Check obproxy1,cff48d57468fbd4f,4098c589aed8968b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:43:10.061 INFO 4486 --- [Check obproxy1,cff48d57468fbd4f,4098c589aed8968b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:43:10.062 INFO 4486 --- [Collect obproxy2,a5075d53d10aad60,edccf94e97d54d6e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:43:10.063 INFO 4486 --- [Collect obproxy1,d76c8aa4291c174a,2e6d04e33b8a632d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:43:10.065 INFO 4486 --- [Collect obproxy1,d76c8aa4291c174a,2e6d04e33b8a632d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:43:10.068 INFO 4486 --- [Collect obproxy1,d76c8aa4291c174a,2e6d04e33b8a632d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:43:10.068 INFO 4486 --- [Collect obproxy2,a5075d53d10aad60,edccf94e97d54d6e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:43:10.611 INFO 4486 --- [ocp-schedules-12,d7161e812d65ea49,86f9eff0a2ad1e18] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:43:10.643 INFO 4486 --- [ocp-schedules-22,62fdb28964d3d884,29dda339e7b4f800] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:43:10.646 WARN 4486 --- [scanner-13273195-1,d7161e812d65ea49,459d8b86bdd2c7db] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:43:10.717 INFO 4486 --- [alarm-detect-8,d7161e812d65ea49,315f7d0df0c2c1c2] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057365 ~ 1729057380 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057380, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057380, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057380=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:43:10.977 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:43:10.994 INFO 4486 --- [prometheus-minute-schedule-1,8880e86586090b93,044987a460a674fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:11.001 INFO 4486 --- [prometheus-minute-schedule-1,8880e86586090b93,044987a460a674fa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:43:11.213 INFO 4486 --- [ocp-async-5,868e0c79681cfd0b,5d708940289dd833] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:11.237 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c1094304597f96ce,1851411956bd2ffc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c1094304597f96ce, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:11.237 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c1094304597f96ce,1851411956bd2ffc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c1094304597f96ce, duration=1 ms] 2024-10-16 13:43:11.582 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,02a8f21b578324e4,2884eec6be40a308] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=02a8f21b578324e4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:11.582 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,02a8f21b578324e4,2884eec6be40a308] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=02a8f21b578324e4, duration=1 ms] 2024-10-16 13:43:12.972 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:13.002 INFO 4486 --- [ocp-async-5,6052b64b321f6d62,9555b23a0758a3f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:43:13.213 INFO 4486 --- [ocp-async-1,1dfc5385ab820cf2,de3e16f1d2549679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:13.654 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057320, nodeCount=1000, affectRows=1000 2024-10-16 13:43:13.659 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057320, nodeCount=999, validCount=999 2024-10-16 13:43:13.692 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057320, nodeCount=1000, affectRows=1000 2024-10-16 13:43:13.697 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057320, nodeCount=1000, affectRows=1000 2024-10-16 13:43:13.700 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057320, nodeCount=1000, validCount=1000 2024-10-16 13:43:13.701 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057320, nodeCount=1000, validCount=1000 2024-10-16 13:43:13.703 ERROR 4486 --- [metric-archive-second-30,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '586774769288417280-1729057320' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:43:13.702812] [YB420AC531B8-00060B7DB03B24F3-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '586774769288417280-1729057320' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:43:13.702812] [YB420AC531B8-00060B7DB03B24F3-0-0] 2024-10-16 13:43:13.708 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057320, nodeCount=6, affectRows=6 2024-10-16 13:43:13.708 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057320, nodeCount=6, validCount=6 2024-10-16 13:43:13.708 ERROR 4486 --- [metric-archive-second-16,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2323956) Duplicate entry '586679390270197760-1729057320' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:43:13.708145] [YB420AC531B8-00060B7DB46FDF1D-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '586679390270197760-1729057320' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:43:13.708145] [YB420AC531B8-00060B7DB46FDF1D-0-0] 2024-10-16 13:43:13.963 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057320, nodeCount=444, affectRows=444 2024-10-16 13:43:13.965 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057320, nodeCount=442, validCount=442 2024-10-16 13:43:13.994 INFO 4486 --- [prometheus-minute-schedule-1,a437cccc798ee4d3,8139aac74cb35205] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:14.003 INFO 4486 --- [ocp-schedules-16,3234c9a3a8f4fa88,dad549622caa66ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:43:14.006 INFO 4486 --- [ocp-schedules-16,556185ea0cb37b4a,64eec5d2c08d0549] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:43:14.006 INFO 4486 --- [prometheus-minute-schedule-1,a437cccc798ee4d3,8139aac74cb35205] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:14.008 INFO 4486 --- [ocp-schedules-16,556185ea0cb37b4a,64eec5d2c08d0549] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:14.040 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.040 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:14.052 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.053 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:14.075 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:43:14.077 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.078 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:14.105 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.106 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.107 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:14.108 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:43:14.108 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.111 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:43:14.112 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:14.121 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.121 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:14.131 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.138 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:14.140 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:43:14.145 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:43:14.150 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:14.151 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:14.172 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:43:14.185 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.187 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.188 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:43:14.230 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:43:14.231 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:43:14.240 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.241 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.242 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:14.246 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.250 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:43:14.251 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:43:14.253 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:14.260 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.260 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.261 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:43:14.270 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:43:14.270 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:43:14.278 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.278 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:14.304 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.304 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:14.327 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.327 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:14.357 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.357 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:14.385 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.385 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:14.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:14.442 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.443 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.444 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:14.448 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:43:14.454 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:43:14.458 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:43:14.460 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:43:14.482 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:43:14.543 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc148042505b1d4e,9de14a990c976691] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=cc148042505b1d4e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:43:14.544 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc148042505b1d4e,9de14a990c976691] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=cc148042505b1d4e, duration=2 ms] 2024-10-16 13:43:14.618 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:43:14.649 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:14.651 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:14.653 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:43:14.684 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:43:14.684 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:43:14.756 INFO 4486 --- [metric-archive-async-22,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057320, nodeCount=480, affectRows=480 2024-10-16 13:43:15.213 INFO 4486 --- [ocp-async-4,a31d4cbe50d00802,55f68039cf4ef5bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:15.411 INFO 4486 --- [metric-archive-async-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057320, nodeCount=480, validCount=480 2024-10-16 13:43:15.973 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:16.994 INFO 4486 --- [prometheus-minute-schedule-1,6d49185cc329a904,750ab274f6a602d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:17.001 INFO 4486 --- [prometheus-minute-schedule-1,6d49185cc329a904,750ab274f6a602d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:17.070 INFO 4486 --- [ocp-schedules-9,9bb1d9885d4c33f5,71de8e8d40302b11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:43:17.150 INFO 4486 --- [ocp-schedules-5,7a0b58086b171c46,6814092ea3b764a2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:43:17.171 INFO 4486 --- [ocp-schedules-18,36436759d8eeccbc,0c217c33f189a594] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:43:17.213 INFO 4486 --- [ocp-async-8,a2ed26dc8cd10343,b9682e746d1493a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:17.506 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a03ebc85668cf427,7f617d5d5ac22464] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a03ebc85668cf427, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:17.506 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a03ebc85668cf427,7f617d5d5ac22464] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a03ebc85668cf427, duration=1 ms] 2024-10-16 13:43:18.974 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:18.996 INFO 4486 --- [ocp-schedules-25,d0a1640014463ebb,517fe7116f8e636b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:43:19.214 INFO 4486 --- [ocp-async-3,e664ed6a698f0d6b,74cb30e0b07394e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:19.543 INFO 4486 --- [ocp-schedules-27,937ee44410a29837,16eadc98055b3a88] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-10-16 13:43:19.545 INFO 4486 --- [ocp-schedules-27,937ee44410a29837,16eadc98055b3a88] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:19.576 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:19.577 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:19.581 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:03:03.102283, 2024-10-16 13:43:19.551, 1000] 2024-10-16 13:43:19.582 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-10-16 02:03:03.102283, endTime 2024-10-16 13:43:19.551 2024-10-16 13:43:19.583 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:42:50.879398, 2024-10-16 13:43:19.551, 1000] 2024-10-16 13:43:19.589 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 5, startTime 2024-10-16 08:42:50.879398, endTime 2024-10-16 13:43:19.551 2024-10-16 13:43:19.589 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:19.594 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:06:51.05999, 2024-10-16 13:43:19.551, 1000] 2024-10-16 13:43:19.598 INFO 4486 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:19.600 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000003, startTime 2024-10-16 02:06:51.05999, endTime 2024-10-16 13:43:19.551 2024-10-16 13:43:19.603 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:19.607 INFO 4486 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:19.609 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:19.628 INFO 4486 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:19.629 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:19.632 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:00:14.755425, 2024-10-16 13:43:19.588, 1000] 2024-10-16 13:43:19.633 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-10-16 08:00:14.755425, endTime 2024-10-16 13:43:19.588 2024-10-16 13:43:19.638 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 12:59:45.930954, 2024-10-16 13:43:19.592, 1000] 2024-10-16 13:43:19.640 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 5, startTime 2024-10-16 12:59:45.930954, endTime 2024-10-16 13:43:19.592 2024-10-16 13:43:19.659 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-12 15:26:31.336743, 2024-10-16 13:43:19.604, 1000] 2024-10-16 13:43:19.664 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000003, startTime 2024-10-12 15:26:31.336743, endTime 2024-10-16 13:43:19.604 2024-10-16 13:43:19.690 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,84bb5e3ff5c0c61d,88434346816e5870] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=84bb5e3ff5c0c61d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:19.690 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,84bb5e3ff5c0c61d,88434346816e5870] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=84bb5e3ff5c0c61d, duration=1 ms] 2024-10-16 13:43:19.994 INFO 4486 --- [prometheus-minute-schedule-1,a6b8c5c40cf64f7e,175d273c7803fdcf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:19.999 INFO 4486 --- [prometheus-minute-schedule-1,a6b8c5c40cf64f7e,175d273c7803fdcf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:20.652 INFO 4486 --- [ocp-schedules-4,f6075362b4c5d0a0,a95ea543498aa32d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:43:20.790 INFO 4486 --- [ocp-schedules-17,45b436de3c4eb9d0,040edfccc8100706] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:43:20.849 WARN 4486 --- [scanner-13274305-1,45b436de3c4eb9d0,0dff1c1f4b31684c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:43:20.996 INFO 4486 --- [prometheus-cache-monitor1,6fb02f32ee285bb5,6036309e5fdd87b0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3414,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:43:20.996 INFO 4486 --- [prometheus-cache-monitor1,6fb02f32ee285bb5,6036309e5fdd87b0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:43:21.214 INFO 4486 --- [ocp-async-9,aa30ff9b364f9187,2d9bbe6d0f4f43d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:21.242 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5d549c0ae815d01b,cfa87d96349758d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5d549c0ae815d01b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:21.242 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5d549c0ae815d01b,cfa87d96349758d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5d549c0ae815d01b, duration=1 ms] 2024-10-16 13:43:21.586 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,60795b828a31fcc4,b5067fa22344272e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=60795b828a31fcc4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:21.586 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,60795b828a31fcc4,b5067fa22344272e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=60795b828a31fcc4, duration=1 ms] 2024-10-16 13:43:21.975 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:22.994 INFO 4486 --- [prometheus-minute-schedule-1,9ff67db0e41ee6f4,1f86deda7566885f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:23.002 INFO 4486 --- [ocp-async-9,ff51d8456d29837c,5e3a1bb81dad4d1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:43:23.002 INFO 4486 --- [prometheus-minute-schedule-1,9ff67db0e41ee6f4,1f86deda7566885f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:23.130 INFO 4486 --- [ocp-schedules-20,9d236599aa7f0c85,548ec3ae3f04b7a7] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:43:23.214 INFO 4486 --- [ocp-async-4,248f5a30bca0b59d,17dc5d0cf32dec0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:24.601 INFO 4486 --- [ocp-schedules-2,96e849cb37b96b05,73ab47095391578f] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:43:24.976 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:25.215 INFO 4486 --- [ocp-async-3,4c4bab0568a5b8dd,5ec5728d8a604a5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:25.307 INFO 4486 --- [ocp-schedules-20,8db22d4ddfbafe77,17bd42f1375b5f61] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:43:25.307 INFO 4486 --- [ocp-schedules-20,8db22d4ddfbafe77,17bd42f1375b5f61] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:43:25.339 INFO 4486 --- [Collect all tenants major compactions1,8db22d4ddfbafe77,cfe547f78a2d526e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:43:25.339 INFO 4486 --- [Collect all tenants major compactions3,8db22d4ddfbafe77,45205f52da5903f3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:43:25.340 INFO 4486 --- [Collect all tenants major compactions5,8db22d4ddfbafe77,a06ff454e1969349] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:43:25.340 INFO 4486 --- [Collect all tenants major compactions2,8db22d4ddfbafe77,46ef948e7aeb3f1f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:43:25.340 INFO 4486 --- [Collect all tenants major compactions4,8db22d4ddfbafe77,ba1ded4b692383e6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:43:25.352 INFO 4486 --- [Collect all tenants major compactions1,8db22d4ddfbafe77,cfe547f78a2d526e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:25.353 INFO 4486 --- [Collect all tenants major compactions1,8db22d4ddfbafe77,cfe547f78a2d526e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:43:25.354 INFO 4486 --- [Collect all tenants major compactions2,8db22d4ddfbafe77,46ef948e7aeb3f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:25.356 INFO 4486 --- [Collect all tenants major compactions2,8db22d4ddfbafe77,46ef948e7aeb3f1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:43:25.357 INFO 4486 --- [Collect all tenants major compactions1,8db22d4ddfbafe77,cfe547f78a2d526e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:43:25.358 INFO 4486 --- [Collect all tenants major compactions1,8db22d4ddfbafe77,cfe547f78a2d526e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:43:25.358 INFO 4486 --- [Collect all tenants major compactions2,8db22d4ddfbafe77,46ef948e7aeb3f1f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:43:25.358 INFO 4486 --- [Collect all tenants major compactions2,8db22d4ddfbafe77,46ef948e7aeb3f1f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:43:25.363 INFO 4486 --- [Collect all tenants major compactions3,8db22d4ddfbafe77,45205f52da5903f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:25.365 INFO 4486 --- [Collect all tenants major compactions3,8db22d4ddfbafe77,45205f52da5903f3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:43:25.371 INFO 4486 --- [Collect all tenants major compactions4,8db22d4ddfbafe77,ba1ded4b692383e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:25.372 INFO 4486 --- [Collect all tenants major compactions5,8db22d4ddfbafe77,a06ff454e1969349] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:25.373 INFO 4486 --- [Collect all tenants major compactions3,8db22d4ddfbafe77,45205f52da5903f3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:43:25.373 INFO 4486 --- [Collect all tenants major compactions3,8db22d4ddfbafe77,45205f52da5903f3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:43:25.373 INFO 4486 --- [Collect all tenants major compactions4,8db22d4ddfbafe77,ba1ded4b692383e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:43:25.373 INFO 4486 --- [Collect all tenants major compactions5,8db22d4ddfbafe77,a06ff454e1969349] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:43:25.376 INFO 4486 --- [Collect all tenants major compactions4,8db22d4ddfbafe77,ba1ded4b692383e6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:43:25.376 INFO 4486 --- [Collect all tenants major compactions4,8db22d4ddfbafe77,ba1ded4b692383e6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:43:25.378 INFO 4486 --- [Collect all tenants major compactions5,8db22d4ddfbafe77,a06ff454e1969349] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:43:25.378 INFO 4486 --- [Collect all tenants major compactions5,8db22d4ddfbafe77,a06ff454e1969349] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:43:25.379 INFO 4486 --- [ocp-schedules-20,8db22d4ddfbafe77,17bd42f1375b5f61] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:43:25.808 INFO 4486 --- [ocp-schedules-28,055f8ae12d96cf60,790bbdd3950e6b21] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:43:25.994 INFO 4486 --- [prometheus-minute-schedule-1,bcc825a6ad3ae303,dee0ff67930cc757] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:26.001 INFO 4486 --- [prometheus-minute-schedule-1,bcc825a6ad3ae303,dee0ff67930cc757] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:27.215 INFO 4486 --- [ocp-async-10,4fb3b2e495c7dab2,d7ac56bac746a949] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:27.240 INFO 4486 --- [ocp-schedules-21,fcab60bcc22f3f71,22377c869d669b1a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:43:27.240+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:43:27.243 INFO 4486 --- [ocp-schedules-21,fcab60bcc22f3f71,22377c869d669b1a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:43:27.244 INFO 4486 --- [ocp-schedules-21,fcab60bcc22f3f71,22377c869d669b1a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:43:27.244 INFO 4486 --- [ocp-schedules-21,fcab60bcc22f3f71,22377c869d669b1a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:43:27.513 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,93cbf39501beecbf,ecdca839721d6489] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=93cbf39501beecbf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:27.514 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,93cbf39501beecbf,ecdca839721d6489] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=93cbf39501beecbf, duration=3 ms] 2024-10-16 13:43:27.977 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:28.994 INFO 4486 --- [prometheus-minute-schedule-1,7d3bf133d9897f8f,5aac2b3d3bd18d5c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:29.000 INFO 4486 --- [prometheus-minute-schedule-1,7d3bf133d9897f8f,5aac2b3d3bd18d5c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:29.215 INFO 4486 --- [ocp-async-6,4da2ca251d644209,3b72b386e6f8305b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:29.665 INFO 4486 --- [ocp-schedules-27,937ee44410a29837,16eadc98055b3a88] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=3, successCount=3 2024-10-16 13:43:29.697 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a761ccee53bc799c,cd588dd619bccbc0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a761ccee53bc799c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:29.697 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a761ccee53bc799c,cd588dd619bccbc0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a761ccee53bc799c, duration=1 ms] 2024-10-16 13:43:30.103 INFO 4486 --- [prometheus-second-schedule-1,55929e7f0668a492,4cbaba5a614e730b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:43:30.659 INFO 4486 --- [ocp-schedules-4,5e8fee45c398c46b,19634cc4ee6657e2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:43:30.978 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:31.009 INFO 4486 --- [ocp-schedules-27,d28519debd7aa8b6,c500d165db6ce4ee] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:43:31.058 WARN 4486 --- [scanner-13274621-1,d28519debd7aa8b6,5d36eb8f56aa8307] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:43:31.084 INFO 4486 --- [alarm-detect-4,d28519debd7aa8b6,b9b2492720277a35] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057380 ~ 1729057401 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057395, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057395, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057395=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:43:31.215 INFO 4486 --- [ocp-async-5,f6de6240d1b8c7ef,99cc2d57c5526525] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:31.247 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9087c3eb4dd33eeb,adf8ba128baf7315] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9087c3eb4dd33eeb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:31.247 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9087c3eb4dd33eeb,adf8ba128baf7315] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9087c3eb4dd33eeb, duration=1 ms] 2024-10-16 13:43:31.590 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d97e10e22e68aca6,94a9e20f9f192f80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d97e10e22e68aca6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:31.590 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d97e10e22e68aca6,94a9e20f9f192f80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d97e10e22e68aca6, duration=1 ms] 2024-10-16 13:43:31.994 INFO 4486 --- [prometheus-minute-schedule-1,5c24fce4f53b4276,3dd493561f9f28b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:32.000 INFO 4486 --- [prometheus-minute-schedule-1,5c24fce4f53b4276,3dd493561f9f28b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:33.002 INFO 4486 --- [ocp-async-5,5cebc1da5985cfc1,4a5ad4beea3810cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:43:33.216 INFO 4486 --- [ocp-async-9,90fd3023ff566146,2f2727d1058c1415] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:33.978 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:34.994 INFO 4486 --- [prometheus-minute-schedule-1,ccf3577b1c0f358e,893a431b90d7a26c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:35.001 INFO 4486 --- [prometheus-minute-schedule-1,ccf3577b1c0f358e,893a431b90d7a26c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:35.216 INFO 4486 --- [ocp-async-8,94bdacc337081a56,30345ecfca3a3305] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:36.980 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:37.063 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:43:37.064 INFO 4486 --- [ocp-schedules-15,f438a4d6800e7e39,6007ae538f385b9d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:43:37.066 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:43:37.066 INFO 4486 --- [ocp-schedules-21,04cd3691f16520d7,7e08e87f59c7bd73] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:43:37.067 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:43:37.069 INFO 4486 --- [ocp-schedules-21,04cd3691f16520d7,7e08e87f59c7bd73] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:37.069 INFO 4486 --- [ocp-schedules-15,f438a4d6800e7e39,6007ae538f385b9d] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:43:37.081 INFO 4486 --- [ocp-schedules-1,56893285e1a4bc60,eb4ba3362a2e01c6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:43:37.083 INFO 4486 --- [ocp-schedules-1,56893285e1a4bc60,eb4ba3362a2e01c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:43:37.089 INFO 4486 --- [ocp-schedules-7,abf0f46966b539b5,2539f856b6564af7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:43:37.090 INFO 4486 --- [ocp-schedules-17,8aa59de93baac5a3,c1b37abcd8dd3b67] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:43:37.091 INFO 4486 --- [ocp-schedules-26,5de0ef816694395a,1561da5bde0bf16d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:43:37.091 INFO 4486 --- [host-check-agent-status1,abf0f46966b539b5,a442499d182b822c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:43:37.091 INFO 4486 --- [host-check-agent-status3,abf0f46966b539b5,dfe18283b94a46c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:43:37.091 INFO 4486 --- [host-check-agent-status2,abf0f46966b539b5,e4c773432a4b8079] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:43:37.092 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:43:37.092 INFO 4486 --- [host-check-agent-status4,abf0f46966b539b5,4ac42fed6d02f6d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:43:37.092 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:43:37.093 INFO 4486 --- [host-check-clockdiff2,5de0ef816694395a,a6ea00c0defdd6c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:43:37.093 INFO 4486 --- [host-check-clockdiff1,5de0ef816694395a,3be9fd2f29ad8c46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:43:37.095 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:43:37.096 INFO 4486 --- [host-check-clockdiff3,5de0ef816694395a,530cf6b3aa1561d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:43:37.096 INFO 4486 --- [host-check-clockdiff4,5de0ef816694395a,31b4eed6b1d22583] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:43:37.097 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:43:37.100 INFO 4486 --- [host-check-agent-status1,abf0f46966b539b5,a442499d182b822c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:37.100 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:37.100 INFO 4486 --- [host-check-agent-status2,abf0f46966b539b5,e4c773432a4b8079] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:43:37.101 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:43:37.102 INFO 4486 --- [host-check-agent-status1,abf0f46966b539b5,a442499d182b822c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:37.102 INFO 4486 --- [host-check-agent-status2,abf0f46966b539b5,e4c773432a4b8079] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:43:37.102 INFO 4486 --- [host-check-agent-status4,abf0f46966b539b5,4ac42fed6d02f6d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:37.103 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:43:37.103 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:37.103 INFO 4486 --- [host-check-agent-status3,abf0f46966b539b5,dfe18283b94a46c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:43:37.104 INFO 4486 --- [host-check-agent-status4,abf0f46966b539b5,4ac42fed6d02f6d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:37.104 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:43:37.105 INFO 4486 --- [host-check-agent-status3,abf0f46966b539b5,dfe18283b94a46c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:43:37.106 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:37.106 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:43:37.108 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:37.129 INFO 4486 --- [host-check-clockdiff1,5de0ef816694395a,3be9fd2f29ad8c46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057417 0 0 , error= 2024-10-16 13:43:37.145 INFO 4486 --- [host-check-clockdiff2,5de0ef816694395a,a6ea00c0defdd6c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057417 4 4 , error= 2024-10-16 13:43:37.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:43:37.162 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.163 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:37.164 INFO 4486 --- [host-check-agent-status1,abf0f46966b539b5,a442499d182b822c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:43:37.166 INFO 4486 --- [host-check-agent-status3,abf0f46966b539b5,dfe18283b94a46c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:43:37.167 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:43:37.167 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:43:37.168 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:43:37.168 INFO 4486 --- [host-check-agent-status2,abf0f46966b539b5,e4c773432a4b8079] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:43:37.168 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:43:37.168 INFO 4486 --- [host-check-agent-status4,abf0f46966b539b5,4ac42fed6d02f6d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:43:37.170 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:43:37.172971471+08:00 2024-10-16 13:43:37.170 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:43:37.169666179+08:00 2024-10-16 13:43:37.170 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:43:37.172983726+08:00 2024-10-16 13:43:37.171 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:43:37.170+08:00 2024-10-16 13:43:37.171 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:43:37.170+08:00 2024-10-16 13:43:37.171 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:43:37.170+08:00 2024-10-16 13:43:37.171 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:43:37.172 INFO 4486 --- [host-check-agent-status1,abf0f46966b539b5,a442499d182b822c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:43:37.172 INFO 4486 --- [host-check-agent-status3,abf0f46966b539b5,dfe18283b94a46c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:43:37.172 INFO 4486 --- [host-check-agent-status1,abf0f46966b539b5,a442499d182b822c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:43:37.172 INFO 4486 --- [host-check-agent-status3,abf0f46966b539b5,dfe18283b94a46c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:43:37.172 INFO 4486 --- [host-check-agent-status1,abf0f46966b539b5,a442499d182b822c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:43:37.172 INFO 4486 --- [host-check-agent-status3,abf0f46966b539b5,dfe18283b94a46c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:43:37.173 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:43:37.176212399+08:00 2024-10-16 13:43:37.173 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:43:37.173+08:00 2024-10-16 13:43:37.173 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:43:37.170+08:00 and status AVAILABLE 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:43:37.170+08:00 and status AVAILABLE 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:43:37.170+08:00 and status AVAILABLE 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-agent-status2,abf0f46966b539b5,e4c773432a4b8079] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-agent-status2,abf0f46966b539b5,e4c773432a4b8079] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-agent-status2,abf0f46966b539b5,e4c773432a4b8079] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-agent-status4,abf0f46966b539b5,4ac42fed6d02f6d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-agent-status4,abf0f46966b539b5,4ac42fed6d02f6d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:43:37.174 INFO 4486 --- [host-check-agent-status4,abf0f46966b539b5,4ac42fed6d02f6d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:43:37.176 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:43:37.173+08:00 and status AVAILABLE 2024-10-16 13:43:37.176 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:43:37.176 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:43:37.176 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:43:37.176 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.178 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:43:37.178 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:43:37.178 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.180 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.181 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.182 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:43:37.183 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:43:37.184 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:37.185 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:43:37.185 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:37.186 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:43:37.186 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:37.186 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:43:37.186 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:37.187 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:43:37.194 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.194 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.194 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.194 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.194 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.195 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.208 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:37.209 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:37.210 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:37.216 INFO 4486 --- [ocp-async-5,526a16e6f0a95d01,458d9e69eb89b3a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:37.216 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:43:37.218 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:43:37.244 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.244 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:43:37.245 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:43:37.245 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:43:37.248 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:43:37.255 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:43:37.258 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.259 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:43:37.259 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:43:37.260 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:43:37.263 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:43:37.268 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.269 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:43:37.272 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.273 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:43:37.274 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:37.275 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:43:37.278 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:37.278 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:37.282 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.283 INFO 4486 --- [ocp-schedules-27,ef64d280e0f896a6,d36e64819e4743ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:43:37.290 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:43:37.306 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:43:37.310 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:43:37.316 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:43:37.320 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:43:37.320 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:43:37.332 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:37.332 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:43:37.339 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:37.350 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:43:37.350 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:37.362 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:43:37.366 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:43:37.369 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:43:37.376 INFO 4486 --- [host-check-host-status4,8aa59de93baac5a3,fef9fa4c0139c833] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:43:37.379 INFO 4486 --- [host-check-host-status3,8aa59de93baac5a3,c8725176d3c2f866] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:43:37.380 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:43:37.392 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:43:37.405 INFO 4486 --- [host-check-host-status2,8aa59de93baac5a3,868bf7cb33f3bb5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:43:37.436 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:43:37.439 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:37.441 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.442 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:43:37.444 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.444 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.445 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:43:37.447 INFO 4486 --- [host-check-host-status1,8aa59de93baac5a3,10f4c258bcdb158e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:43:37.456 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.456 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.456 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.456 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.456 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.456 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.484 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:43:37.521 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:43:37.524 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,64c503f5e5d86e7a,1afc20f90a2bf685] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=64c503f5e5d86e7a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:37.524 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,64c503f5e5d86e7a,1afc20f90a2bf685] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=64c503f5e5d86e7a, duration=3 ms] 2024-10-16 13:43:37.568 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:37.582 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:37.583 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.584 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:43:37.586 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:43:37.586 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.588 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:43:37.589 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.589 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.590 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:43:37.601 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.601 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.601 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.601 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.601 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.602 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.609 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:43:37.792 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:43:37.833 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:37.835 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.836 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:43:37.837 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.837 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.838 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:43:37.848 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.848 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.848 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.848 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.848 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:37.849 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:37.865 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:43:37.964 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:43:37.993 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:43:37.994 INFO 4486 --- [prometheus-minute-schedule-1,914666df4b8f1c0a,b4e920dcc0074305] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:38.002 INFO 4486 --- [prometheus-minute-schedule-1,914666df4b8f1c0a,b4e920dcc0074305] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:38.091 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:43:38.133 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:38.135 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:38.136 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:43:38.137 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:38.137 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:38.138 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:43:38.150 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:38.150 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:38.150 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:38.150 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:38.150 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:43:38.151 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:38.212 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:43:38.418 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:43:38.215997, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:43:38.418 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:43:38.215997, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:43:38.456 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:43:38.215997} } 2024-10-16 13:43:38.471 INFO 4486 --- [ocp-schedules-24,1a6160881fe164e4,46f0dbf23d2b0dff] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:43:38.471 INFO 4486 --- [ocp-schedules-24,1a6160881fe164e4,46f0dbf23d2b0dff] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:43:38.524 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:43:38.531 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:43:38.534 INFO 4486 --- [ocp-schedules-20,3f666ada21a4e73e,18085e15954ffb96] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:43:38.909 INFO 4486 --- [host-check-clockdiff4,5de0ef816694395a,31b4eed6b1d22583] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057418 3 3 , error= 2024-10-16 13:43:39.001 INFO 4486 --- [ocp-schedules-30,ba65c05286847deb,1da0537338fd8b85] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:43:39.216 INFO 4486 --- [ocp-async-9,30c8312fab3977c2,98a39f9fb268e584] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:39.702 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,45337549e938ea3e,eea904ab81fb1f83] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=45337549e938ea3e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:39.702 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,45337549e938ea3e,eea904ab81fb1f83] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=45337549e938ea3e, duration=1 ms] 2024-10-16 13:43:39.842 INFO 4486 --- [host-check-clockdiff3,5de0ef816694395a,530cf6b3aa1561d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057419 4 4 , error= 2024-10-16 13:43:39.981 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:40.006 INFO 4486 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:43:40.006 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:43:40.026 INFO 4486 --- [Check obproxy2,04fa3a16c8c3b0ff,18995350959cc74d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:43:40.026 INFO 4486 --- [Check obproxy1,241478ba7a8bbbd0,828ea75464bb7bf0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:43:40.026 INFO 4486 --- [Check obproxy1,241478ba7a8bbbd0,828ea75464bb7bf0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:40.026 INFO 4486 --- [Check obproxy2,04fa3a16c8c3b0ff,18995350959cc74d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:40.032 INFO 4486 --- [Collect obproxy2,1330dffeab0cbaf6,a3447800c1891dfd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:43:40.032 INFO 4486 --- [Collect obproxy2,1330dffeab0cbaf6,a3447800c1891dfd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:40.034 INFO 4486 --- [Collect obproxy1,e9ed12eb39b0178d,dd70ad2e2ee79db5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:43:40.034 INFO 4486 --- [Collect obproxy1,e9ed12eb39b0178d,dd70ad2e2ee79db5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:43:40.052 INFO 4486 --- [Check obproxy2,04fa3a16c8c3b0ff,18995350959cc74d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:43:40.052 INFO 4486 --- [Check obproxy2,04fa3a16c8c3b0ff,18995350959cc74d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:43:40.054 INFO 4486 --- [Collect obproxy2,1330dffeab0cbaf6,a3447800c1891dfd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:43:40.056 INFO 4486 --- [Collect obproxy1,e9ed12eb39b0178d,dd70ad2e2ee79db5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:43:40.058 INFO 4486 --- [Collect obproxy1,e9ed12eb39b0178d,dd70ad2e2ee79db5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:43:40.059 INFO 4486 --- [Collect obproxy2,1330dffeab0cbaf6,a3447800c1891dfd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:43:40.061 INFO 4486 --- [Check obproxy1,241478ba7a8bbbd0,828ea75464bb7bf0] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:43:40.062 INFO 4486 --- [Check obproxy1,241478ba7a8bbbd0,828ea75464bb7bf0] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:43:40.062 INFO 4486 --- [Collect obproxy1,e9ed12eb39b0178d,dd70ad2e2ee79db5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:43:40.064 INFO 4486 --- [Collect obproxy2,1330dffeab0cbaf6,a3447800c1891dfd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:43:40.666 INFO 4486 --- [ocp-schedules-23,272a7c8f4c8e3ee0,058d1194da423e92] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:43:40.994 INFO 4486 --- [prometheus-minute-schedule-1,263d4c932a35682f,1f87a42114d36255] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:40.995 INFO 4486 --- [prometheus-hour-schedule-1,e9dec57c7bb7fad1,e02a88360dca73ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:43:41.001 INFO 4486 --- [prometheus-minute-schedule-1,263d4c932a35682f,1f87a42114d36255] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:41.139 INFO 4486 --- [ocp-schedules-13,d9eb94acfd4a82bd,c5c3967f6c4454d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:43:41.176 WARN 4486 --- [scanner-13274832-1,d9eb94acfd4a82bd,b8c69650f749197a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:43:41.253 INFO 4486 --- [ocp-async-6,fb94a7285862be5c,489e425f6079ce07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:41.255 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,abe963346b4ace1b,9875aa69aa5f6b3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=abe963346b4ace1b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:41.255 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,abe963346b4ace1b,9875aa69aa5f6b3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=abe963346b4ace1b, duration=2 ms] 2024-10-16 13:43:41.306 INFO 4486 --- [alarm-detect-9,d9eb94acfd4a82bd,5464d6d343ac048f] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057395 ~ 1729057411 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057410, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057410, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057410=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:43:41.594 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4c03ffcd1e30b195,7cf30713f75359bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4c03ffcd1e30b195, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:41.595 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4c03ffcd1e30b195,7cf30713f75359bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4c03ffcd1e30b195, duration=2 ms] 2024-10-16 13:43:41.691 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d46d23161cc987ad,26f5afb0c15bcd72] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=d46d23161cc987ad, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:43:41.692 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d46d23161cc987ad,26f5afb0c15bcd72] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=d46d23161cc987ad, duration=2 ms] 2024-10-16 13:43:42.117 INFO 4486 --- [ocp-schedules-22,871cd96afa1fdce7,f4bf807970aa453e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:43:42.982 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:43.002 INFO 4486 --- [ocp-async-4,27cc463d181b27aa,08caf19e24087046] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:43:43.253 INFO 4486 --- [ocp-async-3,a0880e2501ac9dde,9cc04ed0f80e0fae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:43.516 INFO 4486 --- [ocp-schedules-11,5105bdf2eda545fa,0be785ce40ca59cf] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:43:43.518 INFO 4486 --- [ocp-schedules-11,5105bdf2eda545fa,0be785ce40ca59cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:43.519 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:43:43.519 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:43:43.519 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:43:43.994 INFO 4486 --- [prometheus-minute-schedule-1,97d4b1ced275736d,da3cf7ab4d8db385] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:44.000 INFO 4486 --- [prometheus-minute-schedule-1,97d4b1ced275736d,da3cf7ab4d8db385] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:44.684 INFO 4486 --- [ocp-schedules-16,556185ea0cb37b4a,64eec5d2c08d0549] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:43:44.684 INFO 4486 --- [ocp-schedules-16,9b17c30f6a77956f,c2eca27eacdd5857] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:43:44.687 INFO 4486 --- [ocp-schedules-16,9b17c30f6a77956f,c2eca27eacdd5857] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:43:44.715 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.715 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:44.724 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.724 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:44.752 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:44.753 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.753 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:43:44.758 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.759 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:44.778 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:44.779 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.779 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:44.780 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:43:44.781 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.784 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:43:44.785 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:44.795 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:44.795 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:44.807 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:44.820 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:44.821 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.822 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:44.822 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:43:44.827 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:43:44.832 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:44.833 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:44.859 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:43:44.882 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:44.884 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.885 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:43:44.903 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:43:44.903 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:43:44.945 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:44.948 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.948 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:43:44.949 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:43:44.953 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.956 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:43:44.960 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:44.962 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:44.963 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:44.964 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:43:44.977 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:43:44.977 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:43:44.989 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:44.989 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:45.019 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:45.019 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:45.049 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:45.049 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:45.078 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:45.078 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:43:45.105 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:45.105 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:43:45.137 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:43:45.170 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:45.172 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:45.173 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:43:45.178 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:43:45.184 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:43:45.190 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:43:45.192 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:43:45.216 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:43:45.241 WARN 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : Failed to get ob process info, errMsg:[AgentClient]:http request is failed, response:Cannot get process info of open /proc/1062097/status: no such file or directory, reason: %!v(MISSING) 2024-10-16 13:43:45.246 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 2024-10-16 13:43:45.251 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:45.252 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:45.253 INFO 4486 --- [ocp-async-10,a6521d9350515b7f,8354b3bbb3752930] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:45.275 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:43:45.418 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 got ObProcessInfoResult(pid=91568, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:43:45.446 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:43:45.448 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:43:45.449 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:43:45.476 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:43:45.476 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729056828392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:43:45.498 INFO 4486 --- [ocp-schedules-4,4e60bcf63b1bef51,e2aa2326071ff99a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:43:45.503 INFO 4486 --- [ocp-schedules-4,4e60bcf63b1bef51,e2aa2326071ff99a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:43:45.503 INFO 4486 --- [ocp-schedules-4,4e60bcf63b1bef51,e2aa2326071ff99a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:43:45.519 INFO 4486 --- [ocp-schedules-27,3c8544678aa8e01c,7d3b7ca0b2590fdc] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:43:45.519 INFO 4486 --- [ocp-schedules-27,3c8544678aa8e01c,7d3b7ca0b2590fdc] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:43:45.983 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:46.994 INFO 4486 --- [prometheus-minute-schedule-1,24a71c15cbc2d6c9,89c02ce4f774b146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:47.002 INFO 4486 --- [prometheus-minute-schedule-1,24a71c15cbc2d6c9,89c02ce4f774b146] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:47.070 INFO 4486 --- [ocp-schedules-21,04cd3691f16520d7,7e08e87f59c7bd73] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:43:47.154 INFO 4486 --- [ocp-schedules-30,71a044e8ab71421d,04fa648972161ca0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:43:47.168 INFO 4486 --- [ocp-schedules-1,56893285e1a4bc60,eb4ba3362a2e01c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:43:47.254 INFO 4486 --- [ocp-async-8,d774cb3b113e0a5d,317d5b2ac15040de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:47.530 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,687eab72ad2b805b,dd3775280b182ac7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=687eab72ad2b805b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:47.530 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,687eab72ad2b805b,dd3775280b182ac7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=687eab72ad2b805b, duration=2 ms] 2024-10-16 13:43:48.646 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,334179529cb3864c,bb5dd17422271080] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=334179529cb3864c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:43:48.647 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,334179529cb3864c,bb5dd17422271080] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=334179529cb3864c, duration=2 ms] 2024-10-16 13:43:48.806 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:43:48.984 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:49.254 INFO 4486 --- [ocp-async-2,4b0782fab5bd7b9d,9d7dc9cb470314fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:49.708 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,29da21dd8bf46767,9a30ffc80b0805b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=29da21dd8bf46767, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:49.709 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,29da21dd8bf46767,9a30ffc80b0805b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=29da21dd8bf46767, duration=2 ms] 2024-10-16 13:43:49.994 INFO 4486 --- [prometheus-minute-schedule-1,47493f76c46dda54,efc8eaf683375c64] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:50.002 INFO 4486 --- [prometheus-minute-schedule-1,47493f76c46dda54,efc8eaf683375c64] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:50.439 INFO 4486 --- [ocp-schedules-8,c21ffded529520cf,d5009d7561f89671] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:43:50.439 INFO 4486 --- [ocp-schedules-8,c21ffded529520cf,d5009d7561f89671] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:43:50.441 INFO 4486 --- [ocp-schedules-8,c21ffded529520cf,d5009d7561f89671] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:43:50.441 INFO 4486 --- [ocp-schedules-8,c21ffded529520cf,d5009d7561f89671] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:43:50.441 INFO 4486 --- [ocp-schedules-8,c21ffded529520cf,d5009d7561f89671] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:43:50.441 INFO 4486 --- [ocp-schedules-8,c21ffded529520cf,d5009d7561f89671] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:43:50.517 INFO 4486 --- [ocp-schedules-1,8c342f540e7277e5,ab8e9c963f059d87] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:43:50.517 INFO 4486 --- [ocp-schedules-1,8c342f540e7277e5,ab8e9c963f059d87] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:43:50.517 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:43:50.517 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:43:50.517 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:43:50.517 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:43:50.517 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:43:50.523 INFO 4486 --- [ocp-schedules-1,8c342f540e7277e5,ab8e9c963f059d87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:43:50.523 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:43:50.523 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:43:50.523 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:43:50.523 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:43:50.524 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:43:50.525 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:43:50.525 INFO 4486 --- [ocp-schedules-1,8c342f540e7277e5,ab8e9c963f059d87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:43:50.525 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:43:50.525 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:43:50.525 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:43:50.526 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:43:50.547 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:43:50.547 INFO 4486 --- [ocp-schedules-1,8c342f540e7277e5,ab8e9c963f059d87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:43:50.547 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:43:50.549 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:43:50.550 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:43:50.550 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:43:50.553 INFO 4486 --- [ocp-schedules-1,8c342f540e7277e5,ab8e9c963f059d87] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:43:50.671 INFO 4486 --- [ocp-schedules-18,ee29e7f137807160,a035bd2fcde1a8bf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:43:51.254 INFO 4486 --- [ocp-async-9,3f5cc05124aa6b59,642150f707fa8440] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:51.262 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a4b8b688e2f75769,488cdfa38b6cac3e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a4b8b688e2f75769, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:51.262 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a4b8b688e2f75769,488cdfa38b6cac3e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a4b8b688e2f75769, duration=1 ms] 2024-10-16 13:43:51.437 INFO 4486 --- [ocp-schedules-9,6cef31c0dc74b16c,86eba649cbafcc39] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:43:51.476 WARN 4486 --- [scanner-13275168-1,6cef31c0dc74b16c,280b28e217798f38] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:43:51.601 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,df735c24f85aeee1,1001cab364f1c20c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=df735c24f85aeee1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:51.601 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,df735c24f85aeee1,1001cab364f1c20c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=df735c24f85aeee1, duration=3 ms] 2024-10-16 13:43:51.985 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:52.292 INFO 4486 --- [ocp-schedules-25,8a89fc8c805c1eb6,fc828facd3f5146d] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:43:52.994 INFO 4486 --- [prometheus-minute-schedule-1,9089b46238ce5cbf,b8838c1fd6d6b0e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:53.001 INFO 4486 --- [prometheus-minute-schedule-1,9089b46238ce5cbf,b8838c1fd6d6b0e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:53.003 INFO 4486 --- [ocp-async-1,f738c5c4589f472a,509cdf9680c2c6e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:43:53.254 INFO 4486 --- [ocp-async-8,fd05f6c233e89a0b,1a34e484eec5f7cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:53.519 INFO 4486 --- [ocp-schedules-11,5105bdf2eda545fa,0be785ce40ca59cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:43:54.542 INFO 4486 --- [ocp-schedules-21,1958ef578ef9b11f,24f37a2b56f5a5ac] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:43:54.542+08:00, tenantSize=5 2024-10-16 13:43:54.548 INFO 4486 --- [ocp-schedules-21,1958ef578ef9b11f,24f37a2b56f5a5ac] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729057434543000 2024-10-16 13:43:54.548 INFO 4486 --- [ocp-schedules-21,1958ef578ef9b11f,24f37a2b56f5a5ac] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729057134477000 2024-10-16 13:43:54.582 INFO 4486 --- [ocp-schedules-21,1958ef578ef9b11f,24f37a2b56f5a5ac] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:43:54.588 INFO 4486 --- [ocp-schedules-21,1958ef578ef9b11f,24f37a2b56f5a5ac] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729057434584000 2024-10-16 13:43:54.588 INFO 4486 --- [ocp-schedules-21,1958ef578ef9b11f,24f37a2b56f5a5ac] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729057134516000 2024-10-16 13:43:54.608 INFO 4486 --- [ocp-schedules-21,1958ef578ef9b11f,24f37a2b56f5a5ac] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:43:54.986 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:55.255 INFO 4486 --- [ocp-async-5,4c256c65a7259a5b,13ef8c44dd6a54eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:55.994 INFO 4486 --- [prometheus-minute-schedule-1,3a6d07b86237ceca,1cc78eea5ad4f308] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:56.000 INFO 4486 --- [prometheus-minute-schedule-1,3a6d07b86237ceca,1cc78eea5ad4f308] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:57.255 INFO 4486 --- [ocp-async-9,25dbb36c0088de7b,c22b74a5800b7de6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:57.535 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d9c194fa2654d035,adc07ab5991f5709] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d9c194fa2654d035, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:57.535 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d9c194fa2654d035,adc07ab5991f5709] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d9c194fa2654d035, duration=2 ms] 2024-10-16 13:43:57.988 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:43:58.308 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c6b415b7636ff3dd,a61804098c921335] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=c6b415b7636ff3dd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:43:58.309 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c6b415b7636ff3dd,a61804098c921335] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=c6b415b7636ff3dd, duration=2 ms] 2024-10-16 13:43:58.994 INFO 4486 --- [prometheus-minute-schedule-1,d4c8f23da9024de8,4b007dd47535e58e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057440, interval=60 2024-10-16 13:43:59.002 INFO 4486 --- [prometheus-minute-schedule-1,d4c8f23da9024de8,4b007dd47535e58e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:43:59.005 INFO 4486 --- [ocp-schedules-25,939e31ec837cdec8,d13c4ab769bdcb0c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:43:59.255 INFO 4486 --- [ocp-async-4,33c39c1d61bbec36,a202aa433928654d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:43:59.715 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,fec25f2dbca5cc1c,4495540aecb4b849] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fec25f2dbca5cc1c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:43:59.716 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,fec25f2dbca5cc1c,4495540aecb4b849] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fec25f2dbca5cc1c, duration=2 ms] 2024-10-16 13:44:00.677 INFO 4486 --- [ocp-schedules-4,65a20179c517cfcc,221a859926cf352a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:44:00.988 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:01.255 INFO 4486 --- [ocp-async-7,3df259f5551cb558,9a1fd29918fa3438] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:01.268 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ef13ffd392e51b94,45bd09cd8d4ce461] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ef13ffd392e51b94, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:01.268 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ef13ffd392e51b94,45bd09cd8d4ce461] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ef13ffd392e51b94, duration=1 ms] 2024-10-16 13:44:01.606 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,60d533519caf46d6,874dd4f5ab14157a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=60d533519caf46d6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:01.606 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,60d533519caf46d6,874dd4f5ab14157a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=60d533519caf46d6, duration=1 ms] 2024-10-16 13:44:01.631 INFO 4486 --- [ocp-schedules-13,7ae544f6f58d78c6,28d1352f7392b645] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:44:01.656 WARN 4486 --- [scanner-13275397-1,7ae544f6f58d78c6,70f7b3292fb5551c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:44:01.701 INFO 4486 --- [ocp-schedules-8,1990a1dcc418f1c9,9e36a12cbb931ca0] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:44:01.700+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:44:01.706 INFO 4486 --- [ocp-schedules-8,1990a1dcc418f1c9,9e36a12cbb931ca0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:44:01.724 INFO 4486 --- [ocp-schedules-8,1990a1dcc418f1c9,9e36a12cbb931ca0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:44:01.732 INFO 4486 --- [alarm-detect-19,7ae544f6f58d78c6,955984e1679daa94] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057410 ~ 1729057431 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057425, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057425, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057425=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:44:01.740 INFO 4486 --- [ocp-schedules-8,1990a1dcc418f1c9,9e36a12cbb931ca0] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:44:01.994 INFO 4486 --- [prometheus-minute-schedule-1,f51fcbbe5fb40ffd,6cc6b42a118b7ee0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:01.999 INFO 4486 --- [prometheus-minute-schedule-1,f51fcbbe5fb40ffd,6cc6b42a118b7ee0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:02.222 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:44:02.222 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=7 2024-10-16 13:44:02.222 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1000015 2024-10-16 13:44:02.222 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2024-10-16 13:44:02.222 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2024-10-16 13:44:02.240 WARN 4486 --- [ForkJoinPool.commonPool-worker-19,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:44:02.244 WARN 4486 --- [ForkJoinPool.commonPool-worker-31,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:44:02.256 WARN 4486 --- [ForkJoinPool.commonPool-worker-8,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.289 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 7 agent sql collect property 2024-10-16 13:44:02.290 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:02.294 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:02.298 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:44:02.298 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:44:02.298 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:44:02.299 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2024-10-16 13:44:02.303 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2024-10-16 13:44:02.309 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2024-10-16 13:44:02.311 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:02.315 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:02.315 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:02.317 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:02.320 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1000015 agent sql collect property 2024-10-16 13:44:02.327 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2024-10-16 13:44:02.333 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:02.335 INFO 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 7, do nothing 2024-10-16 13:44:02.336 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:02.347 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313637), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.184), (ob.run.path=/root/oceanbase), (ob.install.path=/root/oceanbase), (ob.data.path=/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=OceanBase), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local ser 2024-10-16 13:44:02.350 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313639), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.178), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data), (monagent.cluster.id=5), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=obce_3zones), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=2), (monagent.obproxy.cluster.name=obce_3zones), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=10.197.49.178), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLev 2024-10-16 13:44:02.366 ERROR 4486 --- [ocp-schedules-24,460e7a1e4edaed80,267685a21adade60] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor1649.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:Unexpected error: key monagent.ob.cgroup.status is not found in config properties! at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:476) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:286) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:114) at com.oceanbase.ocp.executor.executor.AgentExecutor.updateAgentConfig(AgentExecutor.java:193) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentProperties(HostAgentConfigManagerImpl.java:188) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentConfig(HostAgentConfigManagerImpl.java:196) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$FastClassBySpringCGLIB$$798eb4cd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$EnhancerBySpringCGLIB$$eb50b66a.refreshAgentConfig() at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.refreshHostAgentConfig(HostAgentServiceImpl.java:734) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$5f28aa5.refreshHostAgentConfig() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$refreshAgentConfig$0(HostSchedules.java:92) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:44:02.369 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2024-10-16 13:44:02.385 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313644), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=172.28.46.152), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1000003), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=glob_3zone), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=1000002), (monagent.obproxy.cluster.name=xc_OB43), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=172.28.46.152), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition fro 2024-10-16 13:44:02.393 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1000015, do nothing 2024-10-16 13:44:03.003 INFO 4486 --- [ocp-async-2,19ab2223205221b8,dd00f09a8bac9586] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:44:03.255 INFO 4486 --- [ocp-async-9,5cbd8b4121df3ac7,d9434f2999c2aabe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:03.990 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:04.994 INFO 4486 --- [prometheus-minute-schedule-1,a5b3b212720aa1fa,5fd3f6706aeae74b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:05.000 INFO 4486 --- [prometheus-minute-schedule-1,a5b3b212720aa1fa,5fd3f6706aeae74b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:05.256 INFO 4486 --- [ocp-async-4,15ab6429b7243000,faa11c82d21613ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:05.789 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:44:05.789 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:44:05.797 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:44:05.797 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:44:05.797 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:44:05.797 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:44:05.802 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:44:05.802 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:44:05.802 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:44:05.803 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:44:05.868 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:44:05.879 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:44:05.879 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:44:05.879 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:44:05.879 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:44:05.879 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:44:05.887 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:44:06.363 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:44:06.834 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.03192678S 2024-10-16 13:44:06.835 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:44:06.910 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:44:06.910 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:44:06.943 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:44:07.022 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:44:07.022 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:44:07.022 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:44:07.022 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:44:07.022 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:44:07.023 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:44:07.032 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:44:07.066 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:44:07.067 INFO 4486 --- [ocp-schedules-20,f7c7545cc6bfff3e,efdd32e648c7b93a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:44:07.068 INFO 4486 --- [ocp-schedules-20,f7c7545cc6bfff3e,efdd32e648c7b93a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:07.081 INFO 4486 --- [ocp-schedules-29,f40bc3790a0fd599,6cff71b46671ca34] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:44:07.083 INFO 4486 --- [ocp-schedules-29,f40bc3790a0fd599,6cff71b46671ca34] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:44:07.089 INFO 4486 --- [ocp-schedules-1,82feb9de5d2bf0ad,b87e18a46f2947a7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:44:07.090 INFO 4486 --- [ocp-schedules-6,5a328dc8f568a695,f8602aa3964a57a3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:44:07.090 INFO 4486 --- [host-check-clockdiff1,82feb9de5d2bf0ad,13c820493b371887] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:44:07.091 INFO 4486 --- [host-check-agent-status1,5a328dc8f568a695,645324cd103d2ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:44:07.091 INFO 4486 --- [host-check-clockdiff2,82feb9de5d2bf0ad,f225be6faa9ec3d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:44:07.091 INFO 4486 --- [host-check-agent-status2,5a328dc8f568a695,742d09f9d344ac7a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:44:07.092 INFO 4486 --- [host-check-clockdiff3,82feb9de5d2bf0ad,5e0333a93984a4c0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:44:07.092 INFO 4486 --- [host-check-agent-status3,5a328dc8f568a695,b508cfce061e12d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:44:07.092 INFO 4486 --- [host-check-agent-status4,5a328dc8f568a695,39cc0fb8635db84a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:44:07.093 INFO 4486 --- [host-check-clockdiff4,82feb9de5d2bf0ad,10440ae09c8e64b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:44:07.118 INFO 4486 --- [host-check-agent-status2,5a328dc8f568a695,742d09f9d344ac7a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:07.119 INFO 4486 --- [ocp-schedules-13,bd954563a14a64e5,24924f6a83d4602e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:44:07.122 INFO 4486 --- [host-check-agent-status1,5a328dc8f568a695,645324cd103d2ca5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:07.122 INFO 4486 --- [host-check-agent-status3,5a328dc8f568a695,b508cfce061e12d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:07.122 INFO 4486 --- [host-check-clockdiff2,82feb9de5d2bf0ad,f225be6faa9ec3d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057447 4 4 , error= 2024-10-16 13:44:07.122 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:44:07.122 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:44:07.126 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:44:07.127 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:44:07.129 INFO 4486 --- [host-check-clockdiff1,82feb9de5d2bf0ad,13c820493b371887] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057447 0 0 , error= 2024-10-16 13:44:07.129 INFO 4486 --- [host-check-agent-status4,5a328dc8f568a695,39cc0fb8635db84a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:07.129 INFO 4486 --- [host-check-agent-status3,5a328dc8f568a695,b508cfce061e12d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:07.129 INFO 4486 --- [host-check-agent-status1,5a328dc8f568a695,645324cd103d2ca5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:07.130 INFO 4486 --- [host-check-agent-status2,5a328dc8f568a695,742d09f9d344ac7a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:07.131 INFO 4486 --- [host-check-agent-status4,5a328dc8f568a695,39cc0fb8635db84a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:07.134 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:07.135 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:07.136 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:07.136 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:07.137 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:07.137 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:07.138 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:07.140 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.143 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:07.144 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:44:07.156 INFO 4486 --- [host-check-agent-status2,5a328dc8f568a695,742d09f9d344ac7a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:44:07.156 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.157 INFO 4486 --- [host-check-agent-status1,5a328dc8f568a695,645324cd103d2ca5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:44:07.157 INFO 4486 --- [host-check-agent-status4,5a328dc8f568a695,39cc0fb8635db84a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:44:07.157 INFO 4486 --- [host-check-agent-status3,5a328dc8f568a695,b508cfce061e12d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:44:07.157 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:07.161 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status1,5a328dc8f568a695,645324cd103d2ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status3,5a328dc8f568a695,b508cfce061e12d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status3,5a328dc8f568a695,b508cfce061e12d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status1,5a328dc8f568a695,645324cd103d2ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status3,5a328dc8f568a695,b508cfce061e12d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status1,5a328dc8f568a695,645324cd103d2ca5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status4,5a328dc8f568a695,39cc0fb8635db84a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status4,5a328dc8f568a695,39cc0fb8635db84a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status4,5a328dc8f568a695,39cc0fb8635db84a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status2,5a328dc8f568a695,742d09f9d344ac7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status2,5a328dc8f568a695,742d09f9d344ac7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:44:07.163 INFO 4486 --- [host-check-agent-status2,5a328dc8f568a695,742d09f9d344ac7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:44:07.164 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:44:07.164 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:44:07.165 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:44:07.165 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:44:07.165179643+08:00 2024-10-16 13:44:07.165 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:44:07.168844748+08:00 2024-10-16 13:44:07.165 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:44:07.168788967+08:00 2024-10-16 13:44:07.166 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:44:07.165+08:00 2024-10-16 13:44:07.166 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:44:07.165+08:00 2024-10-16 13:44:07.166 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:44:07.166+08:00 2024-10-16 13:44:07.170 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:44:07.171 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:44:07.174486139+08:00 2024-10-16 13:44:07.171 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:44:07.171+08:00 2024-10-16 13:44:07.174 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.176 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:44:07.165+08:00 and status AVAILABLE 2024-10-16 13:44:07.178 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:44:07.166+08:00 and status AVAILABLE 2024-10-16 13:44:07.178 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:44:07.178 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:44:07.179 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:44:07.165+08:00 and status AVAILABLE 2024-10-16 13:44:07.182 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:44:07.182 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:44:07.183 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:44:07.171+08:00 and status AVAILABLE 2024-10-16 13:44:07.185 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:44:07.187 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:07.189 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:07.190 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.191 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:07.191 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:07.191 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:07.192 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:07.192 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:07.193 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:07.194 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:07.200 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.201 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:07.213 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:07.213 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:07.214 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:07.215 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.215 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:07.216 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:07.219 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.220 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:44:07.231 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:07.232 INFO 4486 --- [ocp-schedules-24,a2dbf2b22dda2018,11f39b2640f698f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:44:07.241 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] o.s.boot.SpringApplication : Started application in 1.444 seconds (JVM running for 78350.292) 2024-10-16 13:44:07.241 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.445173801S 2024-10-16 13:44:07.244 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:44:07.245 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:44:07.253 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:44:07.255 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:44:07.255 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:44:07.256 INFO 4486 --- [ocp-async-2,3c1fc203c1f74672,4653bf6d3ba5899a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:07.263 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:44:07.302 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:44:07.305 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:07.307 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:07.309 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:07.313 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:44:07.327 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:07.334 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:44:07.340 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:44:07.344 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:44:07.344 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:44:07.353 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:44:07.355 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:44:07.361 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:07.370 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:07.373 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:07.383 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:44:07.393 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:44:07.401 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:44:07.403 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:44:07.412 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:07.412 INFO 4486 --- [host-check-host-status3,bd954563a14a64e5,64b06aac8dfcc3f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:44:07.415 INFO 4486 --- [host-check-host-status4,bd954563a14a64e5,e01641ae57ba869b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:44:07.417 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:44:07.430 INFO 4486 --- [host-check-host-status2,bd954563a14a64e5,dd62edb7b98e6ad3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:44:07.480 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:44:07.494 INFO 4486 --- [host-check-host-status1,bd954563a14a64e5,7644447a5ee178d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:44:07.540 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,040efcbeb40ad8d0,9aeda8db3409ec3e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=040efcbeb40ad8d0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:07.540 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,040efcbeb40ad8d0,9aeda8db3409ec3e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=040efcbeb40ad8d0, duration=1 ms] 2024-10-16 13:44:07.644 INFO 4486 --- [ocp-schedules-14,1aee02144ba79c30,52b0234a25717959] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:44:07.994 INFO 4486 --- [prometheus-minute-schedule-1,14a6e8aa2d21c0e0,bf712da6f8a353a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:08.000 INFO 4486 --- [prometheus-minute-schedule-1,14a6e8aa2d21c0e0,bf712da6f8a353a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:08.762 INFO 4486 --- [ocp-schedules-26,5b587fa87c0bf35e,41a707efb989411e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:44:08.764 INFO 4486 --- [ocp-schedules-26,5b587fa87c0bf35e,41a707efb989411e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:08.897 INFO 4486 --- [host-check-clockdiff4,82feb9de5d2bf0ad,10440ae09c8e64b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057448 3 3 , error= 2024-10-16 13:44:08.989 INFO 4486 --- [host-check-clockdiff3,82feb9de5d2bf0ad,5e0333a93984a4c0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057448 3 3 , error= 2024-10-16 13:44:09.256 INFO 4486 --- [ocp-async-3,10f0f0072eb9808f,42a5a5c482c40f33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:09.723 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,54880653a37897d2,ba415290a1c5ae37] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=54880653a37897d2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:09.724 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,54880653a37897d2,ba415290a1c5ae37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=54880653a37897d2, duration=2 ms] 2024-10-16 13:44:10.001 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:10.007 INFO 4486 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:44:10.007 INFO 4486 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:44:10.025 INFO 4486 --- [Check obproxy2,6707ab1bcab997cc,fbf4f0117cac916d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:44:10.027 INFO 4486 --- [Check obproxy1,97889f6a56a26d40,5b6f40de13104c0b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:44:10.029 INFO 4486 --- [Check obproxy2,6707ab1bcab997cc,fbf4f0117cac916d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:10.030 INFO 4486 --- [Collect obproxy1,21b4ad1be3d94783,cf7819665f5b46f7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:44:10.030 INFO 4486 --- [Collect obproxy1,21b4ad1be3d94783,cf7819665f5b46f7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:10.031 INFO 4486 --- [Check obproxy1,97889f6a56a26d40,5b6f40de13104c0b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:10.031 INFO 4486 --- [Collect obproxy2,10a83dccb03b47c6,96e229fcdf3bd014] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:44:10.031 INFO 4486 --- [Collect obproxy2,10a83dccb03b47c6,96e229fcdf3bd014] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:10.057 INFO 4486 --- [Check obproxy2,6707ab1bcab997cc,fbf4f0117cac916d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:44:10.057 INFO 4486 --- [Check obproxy2,6707ab1bcab997cc,fbf4f0117cac916d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:44:10.058 INFO 4486 --- [Collect obproxy2,10a83dccb03b47c6,96e229fcdf3bd014] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:44:10.059 INFO 4486 --- [Collect obproxy1,21b4ad1be3d94783,cf7819665f5b46f7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:44:10.063 INFO 4486 --- [Check obproxy1,97889f6a56a26d40,5b6f40de13104c0b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:44:10.063 INFO 4486 --- [Check obproxy1,97889f6a56a26d40,5b6f40de13104c0b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:44:10.064 INFO 4486 --- [Collect obproxy2,10a83dccb03b47c6,96e229fcdf3bd014] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:44:10.065 INFO 4486 --- [Collect obproxy1,21b4ad1be3d94783,cf7819665f5b46f7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:44:10.069 INFO 4486 --- [Collect obproxy2,10a83dccb03b47c6,96e229fcdf3bd014] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:44:10.070 INFO 4486 --- [Collect obproxy1,21b4ad1be3d94783,cf7819665f5b46f7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:44:10.682 INFO 4486 --- [ocp-schedules-30,eb5157cae114bd09,a41190a7f6415de3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:44:10.985 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:44:10.994 INFO 4486 --- [prometheus-minute-schedule-1,a913efd0796160fd,314fbdaad2ab5489] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:11.001 INFO 4486 --- [prometheus-minute-schedule-1,a913efd0796160fd,314fbdaad2ab5489] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:11.007 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:44:11.256 INFO 4486 --- [ocp-async-9,c53bd7c11aefbe60,648fd98248230970] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:11.274 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1a3e7a6bd0d78fd3,09413ebce22921b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1a3e7a6bd0d78fd3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:11.274 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1a3e7a6bd0d78fd3,09413ebce22921b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1a3e7a6bd0d78fd3, duration=1 ms] 2024-10-16 13:44:11.617 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7c7dd0c9fcce3f2c,59481b151c9f4e2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7c7dd0c9fcce3f2c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:11.617 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,7c7dd0c9fcce3f2c,59481b151c9f4e2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7c7dd0c9fcce3f2c, duration=3 ms] 2024-10-16 13:44:11.816 INFO 4486 --- [ocp-schedules-27,3e04373010ccc3c1,1e70027f16be2ed3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:44:11.861 WARN 4486 --- [scanner-13276592-1,3e04373010ccc3c1,dc9795d4e84bae81] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:44:11.897 INFO 4486 --- [alarm-detect-19,3e04373010ccc3c1,11d0b395b29b3411] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057425 ~ 1729057441 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057440, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057440, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057440=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:44:13.003 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:13.003 INFO 4486 --- [ocp-async-6,8c9a05cb05b6da40,beb72ca5db108a5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:44:13.257 INFO 4486 --- [ocp-async-7,23a14d811381be93,718193ccd8e2456f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:13.768 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057380, nodeCount=1000, affectRows=1000 2024-10-16 13:44:13.771 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057380, nodeCount=1000, affectRows=1000 2024-10-16 13:44:13.772 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057380, nodeCount=1000, validCount=1000 2024-10-16 13:44:13.776 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057380, nodeCount=999, validCount=999 2024-10-16 13:44:13.788 ERROR 4486 --- [metric-archive-second-11,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=4117402) Duplicate entry '586774769288417280-1729057380' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:44:13.786565] [YB420AC531B8-00060B7DB4196377-0-0]; nested exception is java.sql.BatchUpdateException: (conn=4117402) Duplicate entry '586774769288417280-1729057380' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:44:13.786565] [YB420AC531B8-00060B7DB4196377-0-0] 2024-10-16 13:44:13.819 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057380, nodeCount=1000, affectRows=1000 2024-10-16 13:44:13.823 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057380, nodeCount=1000, validCount=1000 2024-10-16 13:44:13.994 INFO 4486 --- [prometheus-minute-schedule-1,e6f6ba264a99bcec,fcc954676591a640] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:14.000 INFO 4486 --- [prometheus-minute-schedule-1,e6f6ba264a99bcec,fcc954676591a640] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:14.039 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057380, nodeCount=451, affectRows=451 2024-10-16 13:44:14.041 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057380, nodeCount=449, validCount=449 2024-10-16 13:44:14.779 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057380, nodeCount=477, affectRows=477 2024-10-16 13:44:15.257 INFO 4486 --- [ocp-async-1,38e25af180333e2b,268950bbd202eb05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:15.450 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057380, nodeCount=477, validCount=477 2024-10-16 13:44:15.477 INFO 4486 --- [ocp-schedules-16,9b17c30f6a77956f,c2eca27eacdd5857] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:44:15.477 INFO 4486 --- [ocp-schedules-16,4f5be98006eac6b9,f17564d182428112] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:44:15.479 INFO 4486 --- [ocp-schedules-16,4f5be98006eac6b9,f17564d182428112] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:15.510 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.511 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:15.514 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.515 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:15.545 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.546 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:15.550 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:15.555 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.555 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:44:15.589 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:15.590 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.591 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:15.592 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:44:15.597 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.600 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:44:15.602 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:15.614 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.614 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:15.629 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.639 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:15.639 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.640 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:15.641 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:44:15.646 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:44:15.652 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:15.655 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:15.659 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:15.660 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.661 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:44:15.679 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:44:15.695 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:44:15.695 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:44:15.713 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:15.715 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.716 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:15.720 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.724 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:44:15.727 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:15.760 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:44:15.761 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.761 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:15.771 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:15.776 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.777 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:44:15.787 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:44:15.787 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:44:15.790 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.791 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:15.817 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.817 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:15.845 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.845 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:15.872 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.872 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:15.901 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:15.930 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:15.931 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:15.932 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:15.936 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:44:15.941 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:44:15.947 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:44:15.949 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:44:15.977 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:44:16.004 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:16.115 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:44:16.147 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:16.149 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:16.150 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:44:16.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:44:16.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:44:16.994 INFO 4486 --- [prometheus-minute-schedule-1,9a2b3fc3562681bb,d6352b3f7bf0bc13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:17.002 INFO 4486 --- [prometheus-minute-schedule-1,9a2b3fc3562681bb,d6352b3f7bf0bc13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:17.070 INFO 4486 --- [ocp-schedules-20,f7c7545cc6bfff3e,efdd32e648c7b93a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:44:17.157 INFO 4486 --- [ocp-schedules-25,f3ccea978f517f30,7167cf21d14aed1f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:44:17.162 INFO 4486 --- [ocp-schedules-29,f40bc3790a0fd599,6cff71b46671ca34] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:44:17.257 INFO 4486 --- [ocp-async-10,67c1b28f2ee0fa1c,717e040d36af3d10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:17.546 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,83375fcfea578dc7,cd0b97088960cd4e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=83375fcfea578dc7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:17.546 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,83375fcfea578dc7,cd0b97088960cd4e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=83375fcfea578dc7, duration=2 ms] 2024-10-16 13:44:18.792 INFO 4486 --- [ocp-schedules-26,5b587fa87c0bf35e,41a707efb989411e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:44:19.005 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:19.010 INFO 4486 --- [ocp-schedules-29,d2bdb979f3340dbd,64694c501280991c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:44:19.257 INFO 4486 --- [ocp-async-4,1dad178e6b253889,384fef6bf696a507] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:19.728 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0c8ef32f6257e56a,ed3f27eb7d9bed5b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0c8ef32f6257e56a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:19.728 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0c8ef32f6257e56a,ed3f27eb7d9bed5b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0c8ef32f6257e56a, duration=1 ms] 2024-10-16 13:44:19.994 INFO 4486 --- [prometheus-minute-schedule-1,0bf078bee0ddb83d,707d370a88db49ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:20.000 INFO 4486 --- [prometheus-minute-schedule-1,0bf078bee0ddb83d,707d370a88db49ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:20.688 INFO 4486 --- [ocp-schedules-6,9f04f8c054b789fb,4357e5f4130e1b5f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:44:20.812 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:44:20.996 INFO 4486 --- [prometheus-cache-monitor1,2ef565726fec9471,1eb200f61e1b2c16] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3415,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:44:20.996 INFO 4486 --- [prometheus-cache-monitor1,2ef565726fec9471,1eb200f61e1b2c16] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:44:21.257 INFO 4486 --- [ocp-async-5,17e9184a11e68490,7fed046fd3050068] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:21.280 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,080165a7f605b646,8e689f9f6ae616df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=080165a7f605b646, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:21.280 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,080165a7f605b646,8e689f9f6ae616df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=080165a7f605b646, duration=1 ms] 2024-10-16 13:44:21.621 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f8c0d735d3293166,6919e96c5bf8f106] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f8c0d735d3293166, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:21.622 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f8c0d735d3293166,6919e96c5bf8f106] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f8c0d735d3293166, duration=1 ms] 2024-10-16 13:44:21.976 INFO 4486 --- [ocp-schedules-30,153bc8d1731ebe00,f4e2edf3a1c0edcb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:44:22.006 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:22.007 WARN 4486 --- [scanner-13276767-1,153bc8d1731ebe00,4da8a128341c6b81] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:44:22.994 INFO 4486 --- [prometheus-minute-schedule-1,5e5742a37ee0cd6b,3ee0e3101a81bae7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:23.000 INFO 4486 --- [prometheus-minute-schedule-1,5e5742a37ee0cd6b,3ee0e3101a81bae7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:23.003 INFO 4486 --- [ocp-async-5,0fab343c82e2c66a,b682eeff487bf66c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:44:23.161 INFO 4486 --- [ocp-schedules-19,30398877133a6980,04593c0125298df8] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:44:23.258 INFO 4486 --- [ocp-async-6,40ade88b01699458,232858c21ad9a2e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:24.634 INFO 4486 --- [ocp-schedules-2,cc679cffed96f293,76f90aed39880540] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:44:25.008 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:25.258 INFO 4486 --- [ocp-async-4,714118c05c8c1ee7,8c5b58c2732f8f73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:25.379 INFO 4486 --- [ocp-schedules-20,243db6c110413993,e1cc6db87cc3768e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:44:25.379 INFO 4486 --- [ocp-schedules-20,243db6c110413993,e1cc6db87cc3768e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:44:25.413 INFO 4486 --- [Collect all tenants major compactions1,243db6c110413993,5d43e190647f8db2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:44:25.413 INFO 4486 --- [Collect all tenants major compactions2,243db6c110413993,142d0d5352e18789] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:44:25.414 INFO 4486 --- [Collect all tenants major compactions3,243db6c110413993,92b5e661ba9b0198] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:44:25.414 INFO 4486 --- [Collect all tenants major compactions5,243db6c110413993,a4f934b870f306d6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:44:25.414 INFO 4486 --- [Collect all tenants major compactions4,243db6c110413993,b9f3d3aa8b06428b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:44:25.429 INFO 4486 --- [Collect all tenants major compactions2,243db6c110413993,142d0d5352e18789] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.429 INFO 4486 --- [Collect all tenants major compactions3,243db6c110413993,92b5e661ba9b0198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.430 INFO 4486 --- [Collect all tenants major compactions2,243db6c110413993,142d0d5352e18789] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:44:25.430 INFO 4486 --- [Collect all tenants major compactions1,243db6c110413993,5d43e190647f8db2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.431 INFO 4486 --- [Collect all tenants major compactions3,243db6c110413993,92b5e661ba9b0198] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:44:25.432 INFO 4486 --- [Collect all tenants major compactions1,243db6c110413993,5d43e190647f8db2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:44:25.433 INFO 4486 --- [Collect all tenants major compactions2,243db6c110413993,142d0d5352e18789] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:44:25.434 INFO 4486 --- [Collect all tenants major compactions1,243db6c110413993,5d43e190647f8db2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:44:25.436 INFO 4486 --- [Collect all tenants major compactions3,243db6c110413993,92b5e661ba9b0198] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:44:25.438 INFO 4486 --- [Collect all tenants major compactions2,243db6c110413993,142d0d5352e18789] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:44:25.439 INFO 4486 --- [Collect all tenants major compactions1,243db6c110413993,5d43e190647f8db2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:44:25.441 INFO 4486 --- [Collect all tenants major compactions5,243db6c110413993,a4f934b870f306d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.441 INFO 4486 --- [Collect all tenants major compactions3,243db6c110413993,92b5e661ba9b0198] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:44:25.442 INFO 4486 --- [Collect all tenants major compactions4,243db6c110413993,b9f3d3aa8b06428b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.443 INFO 4486 --- [Collect all tenants major compactions4,243db6c110413993,b9f3d3aa8b06428b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:44:25.443 INFO 4486 --- [Collect all tenants major compactions5,243db6c110413993,a4f934b870f306d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:44:25.446 INFO 4486 --- [Collect all tenants major compactions4,243db6c110413993,b9f3d3aa8b06428b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:44:25.447 INFO 4486 --- [Collect all tenants major compactions4,243db6c110413993,b9f3d3aa8b06428b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:44:25.448 INFO 4486 --- [Collect all tenants major compactions5,243db6c110413993,a4f934b870f306d6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:44:25.448 INFO 4486 --- [Collect all tenants major compactions5,243db6c110413993,a4f934b870f306d6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:44:25.449 INFO 4486 --- [ocp-schedules-20,243db6c110413993,e1cc6db87cc3768e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:44:25.809 INFO 4486 --- [ocp-schedules-15,283c853a4a590d41,2333865f38665a85] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:44:25.810 INFO 4486 --- [ocp-schedules-15,283c853a4a590d41,2333865f38665a85] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:25.854 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:44:25.855 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.857 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:44:25.857 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.859 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:44:25.862 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:44:25.873 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.873 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:44:25.880 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.881 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:44:25.883 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.883 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:44:25.894 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.894 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.894 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:44:25.895 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:44:25.902 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.902 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:44:25.904 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:44:25.905 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.909 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.910 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:44:25.910 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:25.911 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.911 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:25.919 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.920 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:25.921 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.922 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.922 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:44:25.922 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:25.929 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.930 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:44:25.930 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.930 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:44:25.932 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.933 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:44:25.938 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.939 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.939 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:44:25.940 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:44:25.948 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.949 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:44:25.953 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.954 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:44:25.960 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.962 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:25.963 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.963 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.964 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:25.964 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:44:25.964 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.964 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.965 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:25.966 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.966 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:25.967 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:25.967 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.968 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:25.968 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.969 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.969 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:25.970 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.970 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:25.971 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:25.972 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.972 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.972 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:25.973 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.973 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.973 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:25.973 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:44:25.974 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:25.974 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.975 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:25.975 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.976 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.976 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:25.976 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.977 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:25.977 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:25.977 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.978 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:25.979 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:44:25.979 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.980 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:25.982 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.983 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:25.985 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.986 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:25.987 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.987 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:44:25.988 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:25.990 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.991 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:25.993 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.994 INFO 4486 --- [prometheus-minute-schedule-1,ab2a8947ca19b1f0,b4ca764ccd0fb42a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:25.994 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:25.997 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:25.997 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:44:25.998 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:26.000 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:44:26.000 INFO 4486 --- [prometheus-minute-schedule-1,ab2a8947ca19b1f0,b4ca764ccd0fb42a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:26.006 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:26.007 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.007 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:44:26.022 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.023 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:44:26.023 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.023 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:44:26.023 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.023 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:44:26.023 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.023 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:44:26.030 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:44:26.030 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:44:26.038 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.038 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:26.039 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:26.040 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.041 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:44:26.042 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.043 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:26.046 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.047 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:26.050 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.051 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:26.053 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.053 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:44:26.053 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.053 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:44:26.053 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.053 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:44:26.054 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.055 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:26.057 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:44:26.057 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:44:26.057 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.059 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:26.061 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.062 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:44:26.065 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.067 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:44:26.069 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.071 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:44:26.073 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:44:26.075 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:44:26.131 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:26.133 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:26.134 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:44:26.144 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.144 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:44:26.144 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.144 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:44:26.144 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:44:26.144 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:44:26.147 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:44:26.147 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:44:27.258 INFO 4486 --- [ocp-async-5,088e279a639069de,61341fba2ba5368d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:27.275 INFO 4486 --- [ocp-schedules-10,b626cd5884c9ec2b,81f19f91eb5143f2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:44:27.275+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:44:27.278 INFO 4486 --- [ocp-schedules-10,b626cd5884c9ec2b,81f19f91eb5143f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:44:27.279 INFO 4486 --- [ocp-schedules-10,b626cd5884c9ec2b,81f19f91eb5143f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:44:27.280 INFO 4486 --- [ocp-schedules-10,b626cd5884c9ec2b,81f19f91eb5143f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:44:27.553 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2cbab7f8996dd34e,ee834de02ce3ecd7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2cbab7f8996dd34e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:27.553 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2cbab7f8996dd34e,ee834de02ce3ecd7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2cbab7f8996dd34e, duration=1 ms] 2024-10-16 13:44:28.009 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:28.994 INFO 4486 --- [prometheus-minute-schedule-1,6533d14d301a9c0c,4f31773f3e2b88a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:28.999 INFO 4486 --- [prometheus-minute-schedule-1,6533d14d301a9c0c,4f31773f3e2b88a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:29.258 INFO 4486 --- [ocp-async-10,7c2163efc13c4be9,e7d1059dc96106d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:29.733 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f824a379c89a9fc0,fda88232177f61e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f824a379c89a9fc0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:29.733 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f824a379c89a9fc0,fda88232177f61e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f824a379c89a9fc0, duration=1 ms] 2024-10-16 13:44:30.040 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4fff4906e4c0172e,c039b544f3303fe0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=4fff4906e4c0172e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:44:30.041 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4fff4906e4c0172e,c039b544f3303fe0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=4fff4906e4c0172e, duration=2 ms] 2024-10-16 13:44:30.103 INFO 4486 --- [prometheus-second-schedule-1,181ba85600b3401d,d0e4bfdc13ba1bc8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:44:30.693 INFO 4486 --- [ocp-schedules-8,b0153ed8a38ff435,3bccf9dde6548384] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:44:31.010 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:31.259 INFO 4486 --- [ocp-async-9,06ac3a5e66053ff9,6ef784e3bb8578bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:31.286 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,505195189624f752,3db4d1a56a471ef2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=505195189624f752, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:31.287 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,505195189624f752,3db4d1a56a471ef2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=505195189624f752, duration=2 ms] 2024-10-16 13:44:31.630 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a8e3129f6dd64632,1e42479a59b060b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a8e3129f6dd64632, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:31.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a8e3129f6dd64632,1e42479a59b060b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a8e3129f6dd64632, duration=2 ms] 2024-10-16 13:44:31.994 INFO 4486 --- [prometheus-minute-schedule-1,fa7ba126a583209e,f50c5075aaec79f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:32.001 INFO 4486 --- [prometheus-minute-schedule-1,fa7ba126a583209e,f50c5075aaec79f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:32.143 INFO 4486 --- [ocp-schedules-8,9f539bfd74f02383,ddf2115b293f488c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:44:32.184 WARN 4486 --- [scanner-13277124-1,9f539bfd74f02383,a51a9271c2b07c72] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:44:32.246 INFO 4486 --- [alarm-detect-11,9f539bfd74f02383,52ef4010a200896d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057440 ~ 1729057462 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057455, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057455, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057455=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:44:32.250 INFO 4486 --- [alarm-detect-11,9f539bfd74f02383,52ef4010a200896d] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:44:33.004 INFO 4486 --- [ocp-async-8,df67b1b0fa6aa6c2,4f666c4c035a2344] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:44:33.259 INFO 4486 --- [ocp-async-7,493ee33dd3171d0f,e03c0d7627379102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:34.011 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:34.994 INFO 4486 --- [prometheus-minute-schedule-1,e4bc21131c4e780d,f5826953d295cc63] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:35.003 INFO 4486 --- [prometheus-minute-schedule-1,e4bc21131c4e780d,f5826953d295cc63] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:35.259 INFO 4486 --- [ocp-async-3,027941f4ba2d06c1,2ff5522ea08495f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:37.013 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:37.063 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:44:37.064 INFO 4486 --- [ocp-schedules-30,44ec48043da51288,12175a611d87edfd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:44:37.066 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:44:37.066 INFO 4486 --- [ocp-schedules-10,e7fca8566811fe18,3b1d8bcbe3decf79] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:44:37.068 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:44:37.069 INFO 4486 --- [ocp-schedules-10,e7fca8566811fe18,3b1d8bcbe3decf79] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:37.120 INFO 4486 --- [ocp-schedules-19,eec7f26f6a376196,6bd06d688495d1e9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:44:37.121 INFO 4486 --- [ocp-schedules-30,44ec48043da51288,12175a611d87edfd] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:44:37.122 INFO 4486 --- [ocp-schedules-19,eec7f26f6a376196,6bd06d688495d1e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:37.123 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:44:37.129 INFO 4486 --- [ocp-schedules-6,72552b189c2196f8,51d4bec95c16c064] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:44:37.129 INFO 4486 --- [ocp-schedules-7,3992fe131771a7de,613dbbde0312388e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:44:37.129 INFO 4486 --- [ocp-schedules-8,93315927a92fa33a,65d5062fd7b8e47c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:44:37.130 INFO 4486 --- [host-check-clockdiff1,72552b189c2196f8,ba6c5977d7cbfa9b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:44:37.130 INFO 4486 --- [host-check-clockdiff2,72552b189c2196f8,6d44397b985a5a48] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:44:37.130 INFO 4486 --- [host-check-clockdiff3,72552b189c2196f8,c48474a54bfeeb75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:44:37.130 INFO 4486 --- [host-check-clockdiff4,72552b189c2196f8,0128aa73f1e6ab48] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:44:37.130 INFO 4486 --- [host-check-agent-status1,93315927a92fa33a,4943df39d7d98b3c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:44:37.131 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:44:37.132 INFO 4486 --- [host-check-agent-status2,93315927a92fa33a,033a454212f0e3de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:44:37.132 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:44:37.133 INFO 4486 --- [host-check-agent-status3,93315927a92fa33a,ea0a88cc6d5fb1a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:44:37.133 INFO 4486 --- [host-check-agent-status4,93315927a92fa33a,f74b4ea538d71409] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:44:37.133 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:44:37.134 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:44:37.149 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:37.150 INFO 4486 --- [host-check-agent-status2,93315927a92fa33a,033a454212f0e3de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:37.150 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:37.170 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:37.170 INFO 4486 --- [host-check-agent-status1,93315927a92fa33a,4943df39d7d98b3c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:37.170 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:37.150 INFO 4486 --- [host-check-agent-status3,93315927a92fa33a,ea0a88cc6d5fb1a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:37.170 INFO 4486 --- [host-check-agent-status2,93315927a92fa33a,033a454212f0e3de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:37.175 INFO 4486 --- [host-check-agent-status3,93315927a92fa33a,ea0a88cc6d5fb1a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:37.175 INFO 4486 --- [host-check-agent-status1,93315927a92fa33a,4943df39d7d98b3c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:37.176 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:37.176 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:37.176 INFO 4486 --- [host-check-clockdiff2,72552b189c2196f8,6d44397b985a5a48] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057477 4 4 , error= 2024-10-16 13:44:37.176 INFO 4486 --- [host-check-clockdiff1,72552b189c2196f8,ba6c5977d7cbfa9b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057477 0 0 , error= 2024-10-16 13:44:37.179 INFO 4486 --- [host-check-agent-status4,93315927a92fa33a,f74b4ea538d71409] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:37.176 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:37.180 INFO 4486 --- [host-check-clockdiff3,72552b189c2196f8,c48474a54bfeeb75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057477 4 4 , error= 2024-10-16 13:44:37.183 INFO 4486 --- [host-check-agent-status4,93315927a92fa33a,f74b4ea538d71409] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:37.184 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.185 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:37.185 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:37.186 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:44:37.208 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.209 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:37.210 INFO 4486 --- [host-check-agent-status4,93315927a92fa33a,f74b4ea538d71409] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:44:37.211 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:44:37.212 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:44:37.212 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:44:37.214 INFO 4486 --- [host-check-agent-status1,93315927a92fa33a,4943df39d7d98b3c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:44:37.214 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:44:37.215 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:44:37.214318860+08:00 2024-10-16 13:44:37.215 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:44:37.218388462+08:00 2024-10-16 13:44:37.215 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:44:37.218045807+08:00 2024-10-16 13:44:37.215 INFO 4486 --- [host-check-agent-status3,93315927a92fa33a,ea0a88cc6d5fb1a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:44:37.216 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:44:37.215+08:00 2024-10-16 13:44:37.216 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:44:37.215+08:00 2024-10-16 13:44:37.216 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:44:37.215+08:00 2024-10-16 13:44:37.216 INFO 4486 --- [host-check-agent-status2,93315927a92fa33a,033a454212f0e3de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:44:37.217 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:44:37.218 INFO 4486 --- [host-check-agent-status4,93315927a92fa33a,f74b4ea538d71409] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:44:37.218 INFO 4486 --- [host-check-agent-status4,93315927a92fa33a,f74b4ea538d71409] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:44:37.218 INFO 4486 --- [host-check-agent-status4,93315927a92fa33a,f74b4ea538d71409] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:44:37.219 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:44:37.222544784+08:00 2024-10-16 13:44:37.219 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:44:37.219+08:00 2024-10-16 13:44:37.220 INFO 4486 --- [host-check-agent-status3,93315927a92fa33a,ea0a88cc6d5fb1a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:44:37.220 INFO 4486 --- [host-check-agent-status3,93315927a92fa33a,ea0a88cc6d5fb1a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:44:37.220 INFO 4486 --- [host-check-agent-status3,93315927a92fa33a,ea0a88cc6d5fb1a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:44:37.221 INFO 4486 --- [host-check-agent-status1,93315927a92fa33a,4943df39d7d98b3c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:44:37.221 INFO 4486 --- [host-check-agent-status2,93315927a92fa33a,033a454212f0e3de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:44:37.222 INFO 4486 --- [host-check-agent-status1,93315927a92fa33a,4943df39d7d98b3c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:44:37.222 INFO 4486 --- [host-check-agent-status2,93315927a92fa33a,033a454212f0e3de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:44:37.222 INFO 4486 --- [host-check-agent-status1,93315927a92fa33a,4943df39d7d98b3c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:44:37.222 INFO 4486 --- [host-check-agent-status2,93315927a92fa33a,033a454212f0e3de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:44:37.238 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:44:37.215+08:00 and status AVAILABLE 2024-10-16 13:44:37.238 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.238 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.239 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:44:37.240 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:44:37.215+08:00 and status AVAILABLE 2024-10-16 13:44:37.241 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.241 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:44:37.241 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:44:37.219+08:00 and status AVAILABLE 2024-10-16 13:44:37.242 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.242 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:44:37.243 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:44:37.243 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:44:37.243 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:44:37.215+08:00 and status AVAILABLE 2024-10-16 13:44:37.244 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:44:37.246 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:44:37.251 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:37.252 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:37.253 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.253 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:37.253 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:37.254 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:37.255 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:37.255 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:37.255 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:37.256 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:37.259 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.259 INFO 4486 --- [ocp-async-2,4b74be8c2f4bc2b2,6a94a97f3b5bf415] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:37.259 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.259 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.259 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.259 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.260 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.267 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.268 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:37.278 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:44:37.278 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:37.278 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:37.279 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:37.280 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:44:37.281 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.282 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:44:37.285 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.287 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:44:37.297 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.298 INFO 4486 --- [ocp-schedules-1,4f0482efb7b81d9a,3dd3bdb67d7ee31b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:44:37.309 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:44:37.314 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:44:37.318 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:44:37.320 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:44:37.326 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:44:37.328 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:44:37.329 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:44:37.339 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:37.340 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:44:37.346 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:37.348 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:37.357 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:44:37.368 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:44:37.379 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:44:37.380 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:44:37.381 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:44:37.437 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:44:37.443 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:44:37.446 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:44:37.448 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:37.450 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:44:37.463 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:37.464 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:37.467 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:44:37.479 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:44:37.489 INFO 4486 --- [host-check-host-status3,3992fe131771a7de,ac3c58ebfadc7565] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:44:37.497 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:44:37.506 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:44:37.507 INFO 4486 --- [host-check-host-status2,3992fe131771a7de,35f87377472c9d0d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:44:37.517 INFO 4486 --- [host-check-host-status4,3992fe131771a7de,3cc22cd4f0a65ded] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:44:37.527 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:44:37.538 INFO 4486 --- [host-check-host-status1,3992fe131771a7de,e17bc00776fc761b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:44:37.561 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:37.564 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.565 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:44:37.566 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.567 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.568 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:44:37.569 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6ef77669ab65bbf9,891321146214d2f0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6ef77669ab65bbf9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:37.569 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6ef77669ab65bbf9,891321146214d2f0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6ef77669ab65bbf9, duration=3 ms] 2024-10-16 13:44:37.577 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.577 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.577 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.577 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.577 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.577 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.606 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:44:37.641 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:44:37.696 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:37.712 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:37.713 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.714 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:44:37.717 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:44:37.730 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.731 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:44:37.732 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.733 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.734 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:44:37.743 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.743 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.743 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.743 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.743 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.744 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.751 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:44:37.939 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:44:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:44:37.985 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:37.988 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.989 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:44:37.990 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:37.991 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:37.992 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:44:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:44:37.994 INFO 4486 --- [prometheus-minute-schedule-1,7697397d95941392,5f91399e63f5a91b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:38.000 INFO 4486 --- [prometheus-minute-schedule-1,7697397d95941392,5f91399e63f5a91b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:38.003 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.003 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.003 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.003 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.003 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.004 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:38.027 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:44:38.099 INFO 4486 --- [host-check-clockdiff4,72552b189c2196f8,0128aa73f1e6ab48] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057478 3 3 , error= 2024-10-16 13:44:38.251 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:44:38.298 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:38.301 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:38.302 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:44:38.303 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.303 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:38.304 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:44:38.314 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.314 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.314 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.314 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.314 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:44:38.315 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:38.377 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:44:38.443 INFO 4486 --- [ocp-schedules-8,0c71f8635da642ca,debda6188904bc28] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:44:38.444 INFO 4486 --- [ocp-schedules-8,0c71f8635da642ca,debda6188904bc28] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:44:38.444 INFO 4486 --- [ocp-schedules-8,0c71f8635da642ca,debda6188904bc28] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:44:38.579 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:44:38.382606, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:44:38.579 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:44:38.382606, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:44:38.621 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:44:38.382606} } 2024-10-16 13:44:38.678 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:44:38.687 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:44:38.690 INFO 4486 --- [ocp-schedules-4,d0663520e45e11bf,aa47f7b04879054e] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:44:38.856 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a20b5d780257e681,af182034acac1a0c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=a20b5d780257e681, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:44:38.857 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a20b5d780257e681,af182034acac1a0c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=a20b5d780257e681, duration=2 ms] 2024-10-16 13:44:39.014 INFO 4486 --- [ocp-schedules-26,150d8351a8db6fef,51e84c6ebc7dc5f7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:44:39.260 INFO 4486 --- [ocp-async-1,36d66e1ef2b10eeb,d06c795726b6952a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:39.738 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,893687da67fe86bb,0040f358c3dd0668] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=893687da67fe86bb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:39.738 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,893687da67fe86bb,0040f358c3dd0668] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=893687da67fe86bb, duration=1 ms] 2024-10-16 13:44:40.005 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:44:40.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:44:40.015 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:40.025 INFO 4486 --- [Check obproxy2,0aae9d9238003680,db554887fed0da2f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:44:40.026 INFO 4486 --- [Check obproxy2,0aae9d9238003680,db554887fed0da2f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:40.026 INFO 4486 --- [Check obproxy1,b13a1bef5785b97a,08e8f2ce9c951f4b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:44:40.026 INFO 4486 --- [Check obproxy1,b13a1bef5785b97a,08e8f2ce9c951f4b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:40.029 INFO 4486 --- [Collect obproxy2,f78d9cafee94fcf5,b7bc2929b1358592] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:44:40.029 INFO 4486 --- [Collect obproxy2,f78d9cafee94fcf5,b7bc2929b1358592] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:40.033 INFO 4486 --- [Collect obproxy1,d400a9d02bb5b211,bb428792ff8d385c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:44:40.033 INFO 4486 --- [Collect obproxy1,d400a9d02bb5b211,bb428792ff8d385c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:44:40.052 INFO 4486 --- [Collect obproxy2,f78d9cafee94fcf5,b7bc2929b1358592] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:44:40.056 INFO 4486 --- [Check obproxy1,b13a1bef5785b97a,08e8f2ce9c951f4b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:44:40.056 INFO 4486 --- [Check obproxy1,b13a1bef5785b97a,08e8f2ce9c951f4b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:44:40.056 INFO 4486 --- [Check obproxy2,0aae9d9238003680,db554887fed0da2f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:44:40.056 INFO 4486 --- [Check obproxy2,0aae9d9238003680,db554887fed0da2f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:44:40.057 INFO 4486 --- [Collect obproxy2,f78d9cafee94fcf5,b7bc2929b1358592] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:44:40.062 INFO 4486 --- [Collect obproxy2,f78d9cafee94fcf5,b7bc2929b1358592] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:44:40.066 INFO 4486 --- [Collect obproxy1,d400a9d02bb5b211,bb428792ff8d385c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:44:40.069 INFO 4486 --- [Collect obproxy1,d400a9d02bb5b211,bb428792ff8d385c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:44:40.071 INFO 4486 --- [Collect obproxy1,d400a9d02bb5b211,bb428792ff8d385c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:44:40.699 INFO 4486 --- [ocp-schedules-14,732f4308ba196fe1,39d3726576b16a6a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:44:40.994 INFO 4486 --- [prometheus-minute-schedule-1,013a274bd0f93120,fc693a8ff460e1cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:40.995 INFO 4486 --- [prometheus-hour-schedule-1,7ac3d7e71d59d619,46f2207a25d17916] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:44:41.000 INFO 4486 --- [prometheus-minute-schedule-1,013a274bd0f93120,fc693a8ff460e1cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:41.260 INFO 4486 --- [ocp-async-3,bf2cd25380aff348,d5cc63f25b5d1d7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:41.291 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dec5a2e237f19e59,8864fceb5de31f25] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dec5a2e237f19e59, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:41.292 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dec5a2e237f19e59,8864fceb5de31f25] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dec5a2e237f19e59, duration=1 ms] 2024-10-16 13:44:41.636 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9fcebb5f870fd576,7b370f2fa3669599] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9fcebb5f870fd576, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:41.636 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9fcebb5f870fd576,7b370f2fa3669599] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9fcebb5f870fd576, duration=1 ms] 2024-10-16 13:44:42.074 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,57acf0412ac501cb,b8167294a2ab8be1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=57acf0412ac501cb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:44:42.074 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,57acf0412ac501cb,b8167294a2ab8be1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=57acf0412ac501cb, duration=1 ms] 2024-10-16 13:44:42.121 INFO 4486 --- [ocp-schedules-25,c0f0806b9ab0c2d5,e0fbaafc7081eb2a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:44:42.315 INFO 4486 --- [ocp-schedules-27,9e536d104cae27b6,42eb1882f393c031] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:44:42.360 WARN 4486 --- [scanner-13277336-1,9e536d104cae27b6,01b3f686c7651a7a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:44:42.438 INFO 4486 --- [alarm-detect-6,9e536d104cae27b6,e9f84e638017b531] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057455 ~ 1729057472 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057470, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057470, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057470=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:44:42.612 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 13:44:42.612 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 13:44:42.614 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.615 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:44:42.616 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.617 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.618 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.618 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 13:44:42.637 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.637 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.638 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.639 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 13:44:42.653 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.654 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.655 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.655 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 13:44:42.671 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.671 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.672 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.673 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 13:44:42.691 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.692 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.693 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.693 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 13:44:42.700 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.700 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.701 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.701 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 13:44:42.706 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.707 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.707 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.708 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 13:44:42.713 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.714 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.714 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.715 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 13:44:42.721 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.722 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.722 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.723 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 13:44:42.729 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.730 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.730 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.731 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 13:44:42.737 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.737 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.738 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.738 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 13:44:42.748 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.749 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.750 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.750 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 13:44:42.765 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.766 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.767 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.767 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 13:44:42.773 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.773 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.774 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.775 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 13:44:42.780 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.781 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.782 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.782 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 13:44:42.791 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.792 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.794 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.794 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 13:44:42.799 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.800 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.801 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.801 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 13:44:42.807 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.808 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.808 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.809 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 13:44:42.814 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.815 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.815 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.816 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 13:44:42.824 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.825 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.826 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.827 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 13:44:42.833 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.834 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.835 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.835 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 13:44:42.841 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.842 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.842 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.843 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 13:44:42.848 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.848 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.849 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.849 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 13:44:42.854 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.855 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.855 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.856 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 13:44:42.861 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.861 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.862 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.863 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 13:44:42.868 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.868 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.869 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.869 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 13:44:42.875 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.875 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.876 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.877 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 13:44:42.883 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.883 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.884 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.884 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 13:44:42.890 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.890 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.891 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.891 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 13:44:42.896 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.897 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.897 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.898 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 13:44:42.902 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.903 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.903 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.904 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 13:44:42.908 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.909 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.910 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.910 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 13:44:42.916 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.917 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.917 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.918 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 13:44:42.923 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.923 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.924 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.924 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 13:44:42.932 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.933 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.933 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.934 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 13:44:42.941 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.941 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.942 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.942 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 13:44:42.948 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.948 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.949 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.950 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 13:44:42.955 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.955 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.956 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.956 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 13:44:42.960 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.961 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.961 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.962 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 13:44:42.966 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.966 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.967 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.967 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 13:44:42.971 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.972 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.972 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.973 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 13:44:42.977 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.978 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.978 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.979 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 13:44:42.989 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.989 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:42.990 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:42.990 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 13:44:43.004 INFO 4486 --- [ocp-async-6,d351e8d23e3b64fc,b178d9b310e5c517] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:44:43.012 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.013 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.014 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.015 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 13:44:43.016 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:43.021 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.022 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.022 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.023 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 13:44:43.030 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.030 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.031 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.031 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 13:44:43.039 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.040 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.040 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.041 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 13:44:43.045 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.046 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.046 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.047 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 13:44:43.051 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.052 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.052 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.053 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 13:44:43.057 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 13:44:43.057 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 13:44:43.064 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.064 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:44:43.066 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.066 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.067 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.068 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 13:44:43.074 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.074 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:44:43.075 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:43.076 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 13:44:43.080 INFO 4486 --- [ocp-schedules-8,cd3aba8270b71da6,1165104124b6439c] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 13:44:43.260 INFO 4486 --- [ocp-async-9,a0d01ac2509456ee,a1e995a5a2b85232] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:43.994 INFO 4486 --- [prometheus-minute-schedule-1,c48ea2d4f4ac7de2,6e8cd6401e0248b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:44.000 INFO 4486 --- [prometheus-minute-schedule-1,c48ea2d4f4ac7de2,6e8cd6401e0248b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:45.260 INFO 4486 --- [ocp-async-7,baebe66c1c4f09b8,eb1d20e21049689d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:45.504 INFO 4486 --- [ocp-schedules-27,13562178772e5cf2,2af3f04cb349824a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:44:45.510 INFO 4486 --- [ocp-schedules-27,13562178772e5cf2,2af3f04cb349824a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:44:45.510 INFO 4486 --- [ocp-schedules-27,13562178772e5cf2,2af3f04cb349824a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:44:45.524 INFO 4486 --- [ocp-schedules-4,f275c452c3ccf3b6,1f1704051de9d6ea] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:44:45.524 INFO 4486 --- [ocp-schedules-4,f275c452c3ccf3b6,1f1704051de9d6ea] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:44:46.017 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:46.181 INFO 4486 --- [ocp-schedules-16,4f5be98006eac6b9,f17564d182428112] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:44:46.181 INFO 4486 --- [ocp-schedules-16,c902a6dc730fd1a2,d3973d228c03a9e8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:44:46.183 INFO 4486 --- [ocp-schedules-16,c902a6dc730fd1a2,d3973d228c03a9e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:46.211 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.212 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:46.233 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.234 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:46.247 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.248 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.248 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:46.249 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.249 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:44:46.299 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.300 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.301 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:46.302 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:44:46.303 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.306 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:44:46.307 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:46.317 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.318 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:46.330 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.340 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.341 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.342 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:46.342 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:44:46.349 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:44:46.358 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:46.359 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:46.368 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.370 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.371 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:44:46.384 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:44:46.407 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:44:46.407 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:44:46.425 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.427 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.428 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:44:46.432 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.435 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:44:46.439 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:46.462 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:44:46.469 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.469 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:46.474 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.475 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.476 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:44:46.486 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:44:46.487 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:44:46.498 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.498 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:46.520 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.520 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:46.551 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.551 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:44:46.576 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.576 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:44:46.601 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:44:46.627 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.629 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.631 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:44:46.635 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:44:46.641 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:44:46.646 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:44:46.648 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:44:46.674 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:44:46.814 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:44:46.843 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:44:46.845 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:44:46.847 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:44:46.887 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:44:46.887 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:44:46.994 INFO 4486 --- [prometheus-minute-schedule-1,01d30ebc5d01cbac,e962f2a869589384] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:47.000 INFO 4486 --- [prometheus-minute-schedule-1,01d30ebc5d01cbac,e962f2a869589384] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:47.121 INFO 4486 --- [ocp-schedules-10,e7fca8566811fe18,3b1d8bcbe3decf79] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:44:47.159 INFO 4486 --- [ocp-schedules-17,b48da841da811b4d,ffe8660761ce0799] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:44:47.214 INFO 4486 --- [ocp-schedules-19,eec7f26f6a376196,6bd06d688495d1e9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:44:47.260 INFO 4486 --- [ocp-async-10,d53db5ae78da6d10,d3c9905355ba6dc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:47.577 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dccb787c83b2e040,51c28bfd5b4457ec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dccb787c83b2e040, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:47.577 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dccb787c83b2e040,51c28bfd5b4457ec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dccb787c83b2e040, duration=3 ms] 2024-10-16 13:44:49.018 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:49.261 INFO 4486 --- [ocp-async-2,5e71b60705dd1b15,3c0fe464a8398e74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:49.743 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,1c3303249818840b,3d065508b3e4356d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1c3303249818840b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:49.743 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,1c3303249818840b,3d065508b3e4356d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1c3303249818840b, duration=1 ms] 2024-10-16 13:44:49.994 INFO 4486 --- [prometheus-minute-schedule-1,944285393411db8c,03d6fa4da8703798] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:50.003 INFO 4486 --- [prometheus-minute-schedule-1,944285393411db8c,03d6fa4da8703798] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:50.442 INFO 4486 --- [ocp-schedules-17,2867063f5948d37a,36c52883ac1f3d88] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:44:50.442 INFO 4486 --- [ocp-schedules-17,2867063f5948d37a,36c52883ac1f3d88] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:44:50.449 INFO 4486 --- [ocp-schedules-17,2867063f5948d37a,36c52883ac1f3d88] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:44:50.449 INFO 4486 --- [ocp-schedules-17,2867063f5948d37a,36c52883ac1f3d88] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:44:50.453 INFO 4486 --- [ocp-schedules-17,2867063f5948d37a,36c52883ac1f3d88] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:44:50.453 INFO 4486 --- [ocp-schedules-17,2867063f5948d37a,36c52883ac1f3d88] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:44:50.560 INFO 4486 --- [ocp-schedules-18,79282eb0d3b850ab,d38fedde77f23b7f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:44:50.560 INFO 4486 --- [ocp-schedules-18,79282eb0d3b850ab,d38fedde77f23b7f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:44:50.560 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:44:50.560 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:44:50.561 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:44:50.561 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:44:50.561 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:44:50.566 INFO 4486 --- [ocp-schedules-18,79282eb0d3b850ab,d38fedde77f23b7f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:44:50.566 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:44:50.566 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:44:50.566 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:44:50.568 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:44:50.568 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:44:50.568 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:44:50.569 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:44:50.569 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:44:50.569 INFO 4486 --- [ocp-schedules-18,79282eb0d3b850ab,d38fedde77f23b7f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:44:50.570 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:44:50.570 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:44:50.594 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:44:50.594 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:44:50.595 INFO 4486 --- [ocp-schedules-18,79282eb0d3b850ab,d38fedde77f23b7f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:44:50.596 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:44:50.596 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:44:50.596 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:44:50.606 INFO 4486 --- [ocp-schedules-18,79282eb0d3b850ab,d38fedde77f23b7f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:44:50.705 INFO 4486 --- [ocp-schedules-10,44a9bbd031d2cb2c,4693564c242d17be] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:44:51.261 INFO 4486 --- [ocp-async-5,53002c03730ee662,7790ac503b921230] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:51.297 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a9fdc176e59fbf16,edbefdfa010ce551] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a9fdc176e59fbf16, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:51.298 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a9fdc176e59fbf16,edbefdfa010ce551] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a9fdc176e59fbf16, duration=1 ms] 2024-10-16 13:44:51.640 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,91576fbcd64533d4,cadab7b72b619f37] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=91576fbcd64533d4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:51.640 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,91576fbcd64533d4,cadab7b72b619f37] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=91576fbcd64533d4, duration=1 ms] 2024-10-16 13:44:52.019 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:52.503 INFO 4486 --- [ocp-schedules-8,5e5e7f8cd17364a8,893448e7c440f2df] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:44:52.542 WARN 4486 --- [scanner-13277673-1,5e5e7f8cd17364a8,ead1e05ef39d40b8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:44:52.994 INFO 4486 --- [prometheus-minute-schedule-1,6c9b2ad624f3c243,363f34d4b8d06f33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:52.999 INFO 4486 --- [prometheus-minute-schedule-1,6c9b2ad624f3c243,363f34d4b8d06f33] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:53.004 INFO 4486 --- [ocp-async-5,cb4fc07741eb7a09,09ed67c83ac622c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:44:53.261 INFO 4486 --- [ocp-async-3,9e23dba037f65c2d,6ef31cf4a33baa8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:53.520 INFO 4486 --- [ocp-schedules-10,6fd5e252b633d53c,cfde4617ad02c788] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:44:53.521 INFO 4486 --- [ocp-schedules-10,6fd5e252b633d53c,cfde4617ad02c788] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:44:53.522 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:44:53.522 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:44:53.522 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:44:55.020 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:55.261 INFO 4486 --- [ocp-async-4,3c943a28ddb21fa9,9396d6585089d3b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:55.994 INFO 4486 --- [prometheus-minute-schedule-1,d42709f073d5edea,44af158961f2e107] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:56.000 INFO 4486 --- [prometheus-minute-schedule-1,d42709f073d5edea,44af158961f2e107] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:57.262 INFO 4486 --- [ocp-async-5,45fc2dd00f5263e9,b3fd64e3dac41703] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:57.585 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,44bc9ba5629b2945,d4447400aef060bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=44bc9ba5629b2945, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:57.585 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,44bc9ba5629b2945,d4447400aef060bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=44bc9ba5629b2945, duration=1 ms] 2024-10-16 13:44:58.021 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:44:58.994 INFO 4486 --- [prometheus-minute-schedule-1,8936a5b0620f07e3,f7d1559025489bc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057500, interval=60 2024-10-16 13:44:58.999 INFO 4486 --- [prometheus-minute-schedule-1,8936a5b0620f07e3,f7d1559025489bc7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:44:59.019 INFO 4486 --- [ocp-schedules-8,7ed5a0287e09e481,c61c6d56eac00e7d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:44:59.262 INFO 4486 --- [ocp-async-10,9eff9e5069f20dba,91839cde5e843155] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:44:59.749 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,08174bc403118ce4,1b2e369efea3d277] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=08174bc403118ce4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:44:59.749 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,08174bc403118ce4,1b2e369efea3d277] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=08174bc403118ce4, duration=2 ms] 2024-10-16 13:45:00.024 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:40:00.0, nextRun=2024-10-16 13:45:00.0, currentDate=Wed Oct 16 13:45:00 CST 2024 2024-10-16 13:45:00.024 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:40:00.0, nextRunTime=2024-10-16 13:45:00.0} 2024-10-16 13:45:00.028 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-10-16 13:45:00.028 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.197.49.184', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-10-16T13:45:00.028+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='791e68b70ade5cad', operation=EXECUTE} 2024-10-16 13:45:00.045 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:40:00.0, nextRun=2024-10-16 13:45:00.0, currentDate=Wed Oct 16 13:45:00 CST 2024 2024-10-16 13:45:00.045 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:40:00.0, nextRunTime=2024-10-16 13:45:00.0} 2024-10-16 13:45:00.081 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 1000003] 2024-10-16 13:45:00.125 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.p.s.internal.SqlParseTaskSchedule : The last sql parse service is still running, skip current task 2024-10-16 13:45:00.125 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,ea2177639057909f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-10-16 13:45:00.710 INFO 4486 --- [ocp-schedules-13,6bb98624c45fbc7d,71ebc2f9ace005fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:45:01.023 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:01.262 INFO 4486 --- [ocp-async-8,aa8a875c0d866e75,4ed684a244b0d995] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:01.303 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,cf26b9d29e52b855,3f81ced5044a3dd5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cf26b9d29e52b855, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:01.303 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,cf26b9d29e52b855,3f81ced5044a3dd5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cf26b9d29e52b855, duration=1 ms] 2024-10-16 13:45:01.645 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9db82e5abe7a4446,9ae3b9f3833eb09f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9db82e5abe7a4446, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:01.645 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9db82e5abe7a4446,9ae3b9f3833eb09f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9db82e5abe7a4446, duration=1 ms] 2024-10-16 13:45:01.772 INFO 4486 --- [ocp-schedules-21,4e24dcb62e1c3d5f,0e0e127cb247a0ed] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:45:01.772+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:45:01.776 INFO 4486 --- [ocp-schedules-21,4e24dcb62e1c3d5f,0e0e127cb247a0ed] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:45:01.781 INFO 4486 --- [ocp-schedules-21,4e24dcb62e1c3d5f,0e0e127cb247a0ed] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:45:01.793 INFO 4486 --- [ocp-schedules-21,4e24dcb62e1c3d5f,0e0e127cb247a0ed] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:45:01.994 INFO 4486 --- [prometheus-minute-schedule-1,3191ff8b7cd5c889,0a18345857ff827f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:02.000 INFO 4486 --- [prometheus-minute-schedule-1,3191ff8b7cd5c889,0a18345857ff827f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:02.760 INFO 4486 --- [ocp-schedules-7,126c06869f75fa40,76ad72d9f525bee2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:45:02.783 WARN 4486 --- [scanner-13277889-1,126c06869f75fa40,a6e3ef077e27df3d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:45:02.896 INFO 4486 --- [alarm-detect-8,126c06869f75fa40,fb2c5fbb1dab0f87] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057470 ~ 1729057492 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057485, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057485, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057485=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:45:03.004 INFO 4486 --- [ocp-async-2,2d94d8dea56906f0,d71602cc03b02721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:45:03.262 INFO 4486 --- [ocp-async-5,1f3d57e1b8787f17,02ca72d1a075587e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:03.523 INFO 4486 --- [ocp-schedules-10,6fd5e252b633d53c,cfde4617ad02c788] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:45:04.024 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:04.994 INFO 4486 --- [prometheus-minute-schedule-1,8b7d03e08d9153b6,1214dbf4b66bdc41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:05.000 INFO 4486 --- [prometheus-minute-schedule-1,8b7d03e08d9153b6,1214dbf4b66bdc41] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:05.262 INFO 4486 --- [ocp-async-6,3032232245aa317f,9b8847bde26382bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:06.070 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,17a10deddec7f767,70c7c105e045824d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=17a10deddec7f767, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 13:45:06.295 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,17a10deddec7f767,70c7c105e045824d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=17a10deddec7f767, duration=239 ms] 2024-10-16 13:45:07.025 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:07.066 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:45:07.067 INFO 4486 --- [ocp-schedules-2,282269bd89165e8e,0322200239d53043] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:45:07.068 INFO 4486 --- [ocp-schedules-2,282269bd89165e8e,0322200239d53043] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:45:07.081 INFO 4486 --- [ocp-schedules-25,1787c9edee704848,43cf4ff612c61774] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:45:07.083 INFO 4486 --- [ocp-schedules-25,1787c9edee704848,43cf4ff612c61774] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:45:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:45:07.089 INFO 4486 --- [ocp-schedules-20,65d7630c8a2d907b,e7891d12e93a0549] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:45:07.090 INFO 4486 --- [ocp-schedules-3,3974e2a56df72ab9,203ad1bd90543bf1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:45:07.090 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:45:07.090 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:45:07.091 INFO 4486 --- [host-check-agent-status1,3974e2a56df72ab9,4a91c92d01704495] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:45:07.091 INFO 4486 --- [host-check-agent-status2,3974e2a56df72ab9,b239f8e28c2236a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:45:07.091 INFO 4486 --- [host-check-agent-status4,3974e2a56df72ab9,0fd5260340528cc2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:45:07.091 INFO 4486 --- [ocp-schedules-26,bff18cd53ecda70e,89877ee20c69ca7c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:45:07.091 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:45:07.091 INFO 4486 --- [host-check-agent-status3,3974e2a56df72ab9,6a64608b47687b50] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:45:07.092 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:45:07.092 INFO 4486 --- [host-check-clockdiff2,bff18cd53ecda70e,b9daf6d3e74fa8d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:45:07.092 INFO 4486 --- [host-check-clockdiff1,bff18cd53ecda70e,541e9ea984db1609] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:45:07.095 INFO 4486 --- [host-check-clockdiff4,bff18cd53ecda70e,0fa577690bfb75cb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:45:07.095 INFO 4486 --- [host-check-clockdiff3,bff18cd53ecda70e,3bc5076c4cd13a50] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:45:07.098 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:45:07.098 INFO 4486 --- [host-check-agent-status3,3974e2a56df72ab9,6a64608b47687b50] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:45:07.100 INFO 4486 --- [host-check-agent-status4,3974e2a56df72ab9,0fd5260340528cc2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:45:07.100 INFO 4486 --- [host-check-agent-status1,3974e2a56df72ab9,4a91c92d01704495] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:07.100 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:45:07.101 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:45:07.102 INFO 4486 --- [host-check-agent-status2,3974e2a56df72ab9,b239f8e28c2236a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:45:07.103 INFO 4486 --- [host-check-agent-status1,3974e2a56df72ab9,4a91c92d01704495] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:07.106 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:07.103 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:45:07.103 INFO 4486 --- [host-check-agent-status3,3974e2a56df72ab9,6a64608b47687b50] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:45:07.109 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:45:07.110 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:45:07.110 INFO 4486 --- [host-check-agent-status4,3974e2a56df72ab9,0fd5260340528cc2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:45:07.111 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:07.117 INFO 4486 --- [host-check-agent-status2,3974e2a56df72ab9,b239f8e28c2236a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:45:07.117 INFO 4486 --- [host-check-clockdiff4,bff18cd53ecda70e,0fa577690bfb75cb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057507 3 3 , error= 2024-10-16 13:45:07.122 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.136 INFO 4486 --- [host-check-clockdiff1,bff18cd53ecda70e,541e9ea984db1609] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057507 0 0 , error= 2024-10-16 13:45:07.134 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:07.139 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:45:07.157 INFO 4486 --- [host-check-agent-status2,3974e2a56df72ab9,b239f8e28c2236a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:45:07.157 INFO 4486 --- [host-check-agent-status1,3974e2a56df72ab9,4a91c92d01704495] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:45:07.158 INFO 4486 --- [host-check-agent-status3,3974e2a56df72ab9,6a64608b47687b50] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:45:07.159 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:45:07.166 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:45:07.168 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:45:07.170922366+08:00 2024-10-16 13:45:07.168 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.168 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:45:07.168+08:00 2024-10-16 13:45:07.169 INFO 4486 --- [host-check-agent-status3,3974e2a56df72ab9,6a64608b47687b50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:45:07.169 INFO 4486 --- [host-check-agent-status3,3974e2a56df72ab9,6a64608b47687b50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:45:07.169 INFO 4486 --- [host-check-agent-status3,3974e2a56df72ab9,6a64608b47687b50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:45:07.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:07.169 INFO 4486 --- [host-check-agent-status2,3974e2a56df72ab9,b239f8e28c2236a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:45:07.169 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:45:07.172399587+08:00 2024-10-16 13:45:07.169 INFO 4486 --- [host-check-agent-status2,3974e2a56df72ab9,b239f8e28c2236a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:45:07.170 INFO 4486 --- [host-check-agent-status2,3974e2a56df72ab9,b239f8e28c2236a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:45:07.170 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:45:07.170 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:45:07.170+08:00 2024-10-16 13:45:07.171 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:45:07.171 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:45:07.174747875+08:00 2024-10-16 13:45:07.171 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:45:07.168+08:00 and status AVAILABLE 2024-10-16 13:45:07.171 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:45:07.171+08:00 2024-10-16 13:45:07.172 INFO 4486 --- [host-check-agent-status4,3974e2a56df72ab9,0fd5260340528cc2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:45:07.172 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:45:07.170+08:00 and status AVAILABLE 2024-10-16 13:45:07.173 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:45:07.173 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:45:07.173103221+08:00 2024-10-16 13:45:07.174 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:45:07.174 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:45:07.173+08:00 2024-10-16 13:45:07.174 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:45:07.171+08:00 and status AVAILABLE 2024-10-16 13:45:07.175 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:45:07.176 INFO 4486 --- [host-check-agent-status4,3974e2a56df72ab9,0fd5260340528cc2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:45:07.176 INFO 4486 --- [host-check-agent-status4,3974e2a56df72ab9,0fd5260340528cc2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:45:07.176 INFO 4486 --- [host-check-agent-status4,3974e2a56df72ab9,0fd5260340528cc2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:45:07.176 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:45:07.173+08:00 and status AVAILABLE 2024-10-16 13:45:07.176 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:45:07.178 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:45:07.180 INFO 4486 --- [host-check-agent-status1,3974e2a56df72ab9,4a91c92d01704495] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:45:07.180 INFO 4486 --- [host-check-agent-status1,3974e2a56df72ab9,4a91c92d01704495] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:45:07.180 INFO 4486 --- [host-check-agent-status1,3974e2a56df72ab9,4a91c92d01704495] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:45:07.181 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.183 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:45:07.184 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:45:07.185 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:45:07.185 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:45:07.185 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:45:07.186 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:45:07.186 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:45:07.187 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:07.188 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:07.194 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.195 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:45:07.204 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.205 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:45:07.208 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:07.209 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:07.210 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:07.210 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:07.214 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.215 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:45:07.218 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.219 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:45:07.227 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:07.229 INFO 4486 --- [ocp-schedules-19,d97c42b334c21d7b,4d728a2257296b34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:45:07.240 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:45:07.243 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:45:07.250 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:45:07.250 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:45:07.254 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:45:07.260 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:45:07.261 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:45:07.263 INFO 4486 --- [ocp-async-8,29c9eaad0a036305,7ffaf1e43122ac2b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:07.269 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:07.269 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:07.270 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:45:07.280 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:07.290 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:07.299 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:45:07.303 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:45:07.312 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:45:07.314 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:45:07.317 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:45:07.328 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:45:07.332 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:07.334 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:07.348 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:07.354 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:45:07.363 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:45:07.365 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:45:07.376 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:45:07.379 INFO 4486 --- [host-check-host-status3,65d7630c8a2d907b,9040ad95920cfefb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:45:07.381 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:07.387 INFO 4486 --- [host-check-host-status4,65d7630c8a2d907b,b22cc9c003a89cd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:45:07.387 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:45:07.400 INFO 4486 --- [host-check-host-status2,65d7630c8a2d907b,7be0e3fddbb91f46] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:45:07.432 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:45:07.442 INFO 4486 --- [host-check-host-status1,65d7630c8a2d907b,454c9d30202ef85a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:45:07.590 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,07a924dc365462a1,acd513c0e696f443] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=07a924dc365462a1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:07.590 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,07a924dc365462a1,acd513c0e696f443] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=07a924dc365462a1, duration=1 ms] 2024-10-16 13:45:07.994 INFO 4486 --- [prometheus-minute-schedule-1,56eba1775adef08d,a2e164adf2cea4fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:08.000 INFO 4486 --- [prometheus-minute-schedule-1,56eba1775adef08d,a2e164adf2cea4fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:08.060 INFO 4486 --- [host-check-clockdiff3,bff18cd53ecda70e,3bc5076c4cd13a50] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057508 3 3 , error= 2024-10-16 13:45:08.066 INFO 4486 --- [host-check-clockdiff2,bff18cd53ecda70e,b9daf6d3e74fa8d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057508 4 4 , error= 2024-10-16 13:45:09.263 INFO 4486 --- [ocp-async-1,e584690d746c7222,00d0d67f8c5a548d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:09.753 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dd05afcdfebf7260,1a5ab87bef0d4ec1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dd05afcdfebf7260, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:09.753 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dd05afcdfebf7260,1a5ab87bef0d4ec1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dd05afcdfebf7260, duration=1 ms] 2024-10-16 13:45:10.029 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:10.034 INFO 4486 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:45:10.034 INFO 4486 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:45:10.042 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:45:10.056 INFO 4486 --- [Check obproxy2,c7f2ea284b56226f,f19240dcc5e84967] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:45:10.056 INFO 4486 --- [Check obproxy2,c7f2ea284b56226f,f19240dcc5e84967] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:10.057 INFO 4486 --- [Check obproxy1,f87c11f8b7a0726e,e8d8626d6eca6e2f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:45:10.057 INFO 4486 --- [Check obproxy1,f87c11f8b7a0726e,e8d8626d6eca6e2f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:10.060 INFO 4486 --- [Collect obproxy1,7e33b3ece0624000,a677f87e90151475] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:45:10.060 INFO 4486 --- [Collect obproxy1,7e33b3ece0624000,a677f87e90151475] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:10.061 INFO 4486 --- [Collect obproxy2,61fcbf1fa8ffbb7f,237c15b173f93b02] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:45:10.061 INFO 4486 --- [Collect obproxy2,61fcbf1fa8ffbb7f,237c15b173f93b02] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:10.083 INFO 4486 --- [Check obproxy1,f87c11f8b7a0726e,e8d8626d6eca6e2f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:45:10.083 INFO 4486 --- [Collect obproxy2,61fcbf1fa8ffbb7f,237c15b173f93b02] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:45:10.083 INFO 4486 --- [Check obproxy1,f87c11f8b7a0726e,e8d8626d6eca6e2f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:45:10.083 INFO 4486 --- [Collect obproxy1,7e33b3ece0624000,a677f87e90151475] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:45:10.084 INFO 4486 --- [Check obproxy2,c7f2ea284b56226f,f19240dcc5e84967] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:45:10.084 INFO 4486 --- [Check obproxy2,c7f2ea284b56226f,f19240dcc5e84967] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:45:10.089 INFO 4486 --- [Collect obproxy2,61fcbf1fa8ffbb7f,237c15b173f93b02] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:45:10.089 INFO 4486 --- [Collect obproxy1,7e33b3ece0624000,a677f87e90151475] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:45:10.092 INFO 4486 --- [Collect obproxy1,7e33b3ece0624000,a677f87e90151475] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:45:10.093 INFO 4486 --- [Collect obproxy2,61fcbf1fa8ffbb7f,237c15b173f93b02] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:45:10.716 INFO 4486 --- [ocp-schedules-13,a0531185f6aacc2c,16317b0b1d76316a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:45:10.978 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:45:10.994 INFO 4486 --- [prometheus-minute-schedule-1,f0d551fe73c2943e,506310dea6aa1903] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:11.004 INFO 4486 --- [prometheus-minute-schedule-1,f0d551fe73c2943e,506310dea6aa1903] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:11.007 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 23, ruleSize = 13. 2024-10-16 13:45:11.263 INFO 4486 --- [ocp-async-7,e764ed5b0fe60626,6850b06046903525] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:11.308 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,22f526a1b8eeae5b,66a9deaa6175a332] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=22f526a1b8eeae5b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:11.308 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,22f526a1b8eeae5b,66a9deaa6175a332] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=22f526a1b8eeae5b, duration=1 ms] 2024-10-16 13:45:11.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ddb424dcc43299e7,a451dc59bbcf10d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=ddb424dcc43299e7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:45:11.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ddb424dcc43299e7,a451dc59bbcf10d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=ddb424dcc43299e7, duration=1 ms] 2024-10-16 13:45:11.649 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c88105039e995650,f928980b11086bf9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c88105039e995650, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:11.649 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c88105039e995650,f928980b11086bf9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c88105039e995650, duration=1 ms] 2024-10-16 13:45:12.973 INFO 4486 --- [ocp-schedules-5,40fa106c32326909,53da46274233e29a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:45:12.993 WARN 4486 --- [scanner-13279847-1,40fa106c32326909,62e3b92021798375] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:45:13.005 INFO 4486 --- [ocp-async-10,43ce5313046f177f,81036dbee1056147] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:45:13.030 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:13.090 INFO 4486 --- [alarm-detect-15,40fa106c32326909,db03b512b1c242bd] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057485 ~ 1729057503 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057500, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057500, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057500=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:45:13.263 INFO 4486 --- [ocp-async-3,4fad217f6ac91438,5d99552948c7cdaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:13.382 INFO 4486 --- [ocp-schedules-28,b8cb28471a997576,a42495189419fae8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 13:45:13.382 INFO 4486 --- [ocp-schedules-28,b8cb28471a997576,a42495189419fae8] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 13:45:13.417 INFO 4486 --- [pool-280-thread-1,b8cb28471a997576,7ff11ef4848b737b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:45:13.417 INFO 4486 --- [pool-280-thread-2,b8cb28471a997576,68ce64cd766a0d80] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:45:13.417 INFO 4486 --- [pool-280-thread-3,b8cb28471a997576,696c23506ed2814d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:45:13.436 INFO 4486 --- [pool-280-thread-1,b8cb28471a997576,7ff11ef4848b737b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:13.437 INFO 4486 --- [pool-280-thread-2,b8cb28471a997576,68ce64cd766a0d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:13.439 INFO 4486 --- [pool-280-thread-1,b8cb28471a997576,7ff11ef4848b737b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057513000000] 2024-10-16 13:45:13.441 INFO 4486 --- [pool-280-thread-2,b8cb28471a997576,68ce64cd766a0d80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057513000000] 2024-10-16 13:45:13.449 INFO 4486 --- [pool-280-thread-3,b8cb28471a997576,696c23506ed2814d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:13.453 INFO 4486 --- [pool-280-thread-3,b8cb28471a997576,696c23506ed2814d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057513000000] 2024-10-16 13:45:13.729 WARN 4486 --- [pool-280-thread-3,b8cb28471a997576,696c23506ed2814d] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 13:45:13.729 WARN 4486 --- [pool-280-thread-3,b8cb28471a997576,696c23506ed2814d] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 13:45:13.731 INFO 4486 --- [pool-280-thread-3,b8cb28471a997576,696c23506ed2814d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:45:13.860 ERROR 4486 --- [metric-archive-second-29,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2323956) Duplicate entry '587073993980841984-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.859592] [YB420AC531B8-00060B7DB7688580-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '587073993980841984-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.859592] [YB420AC531B8-00060B7DB7688580-0-0] 2024-10-16 13:45:13.860 ERROR 4486 --- [metric-archive-second-14,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '586679390261809152-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.859964] [YB420AC531B8-00060B7DB39877EB-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '586679390261809152-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.859964] [YB420AC531B8-00060B7DB39877EB-0-0] 2024-10-16 13:45:13.860 ERROR 4486 --- [metric-archive-second-23,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '586774769053536256-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.859851] [YB420AC531B8-00060B7DB03B254D-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '586774769053536256-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.859851] [YB420AC531B8-00060B7DB03B254D-0-0] 2024-10-16 13:45:13.866 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=1000, affectRows=1000 2024-10-16 13:45:13.872 ERROR 4486 --- [metric-archive-second-22,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=835727) Duplicate entry '587073990931582977-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.860539] [YB420AC531B8-00060B7DB46FEAF8-0-0]; nested exception is java.sql.BatchUpdateException: (conn=835727) Duplicate entry '587073990931582977-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.860539] [YB420AC531B8-00060B7DB46FEAF8-0-0] 2024-10-16 13:45:13.872 ERROR 4486 --- [metric-archive-second-32,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620710) Duplicate entry '587073990944165888-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.863008] [YB420AC531B8-00060B7DB41963B7-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620710) Duplicate entry '587073990944165888-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.863008] [YB420AC531B8-00060B7DB41963B7-0-0] 2024-10-16 13:45:13.872 ERROR 4486 --- [metric-archive-second-19,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '669741696503320576-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.866901] [YB420AC531B8-00060B7DBC3872B6-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '669741696503320576-1729057440' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:13.866901] [YB420AC531B8-00060B7DBC3872B6-0-0] 2024-10-16 13:45:13.879 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=999, validCount=999 2024-10-16 13:45:13.884 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=1000, affectRows=1000 2024-10-16 13:45:13.888 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=1000, validCount=1000 2024-10-16 13:45:13.916 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=61, affectRows=61 2024-10-16 13:45:13.917 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=61, validCount=61 2024-10-16 13:45:13.975 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=140, affectRows=140 2024-10-16 13:45:13.976 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=140, validCount=140 2024-10-16 13:45:13.994 INFO 4486 --- [prometheus-minute-schedule-1,cd7165c45be653ec,86a6ca6d96913925] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:14.000 INFO 4486 --- [prometheus-minute-schedule-1,cd7165c45be653ec,86a6ca6d96913925] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:14.005 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=176, affectRows=176 2024-10-16 13:45:14.006 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=175, validCount=175 2024-10-16 13:45:14.022 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=204, affectRows=204 2024-10-16 13:45:14.023 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=203, validCount=203 2024-10-16 13:45:14.061 WARN 4486 --- [pool-280-thread-1,b8cb28471a997576,7ff11ef4848b737b] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 13:45:14.061 WARN 4486 --- [pool-280-thread-1,b8cb28471a997576,7ff11ef4848b737b] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1009 2024-10-16 13:45:14.062 INFO 4486 --- [pool-280-thread-1,b8cb28471a997576,7ff11ef4848b737b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:45:14.099 INFO 4486 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=350, affectRows=350 2024-10-16 13:45:14.101 INFO 4486 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=350, validCount=350 2024-10-16 13:45:14.157 WARN 4486 --- [pool-280-thread-2,b8cb28471a997576,68ce64cd766a0d80] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 13:45:14.158 INFO 4486 --- [pool-280-thread-2,b8cb28471a997576,68ce64cd766a0d80] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:45:14.159 INFO 4486 --- [ocp-schedules-28,b8cb28471a997576,a42495189419fae8] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 13:45:14.173 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057440, nodeCount=521, affectRows=521 2024-10-16 13:45:14.175 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057440, nodeCount=521, validCount=521 2024-10-16 13:45:14.791 INFO 4486 --- [metric-archive-async-20,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057440, nodeCount=4, affectRows=4 2024-10-16 13:45:14.795 ERROR 4486 --- [metric-archive-async-21,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=620710) Duplicate entry '662413617133735936-1729057440-700500076079419392' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.794417] [YB420AC531B8-00060B7DAD2B73E0-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620710) Duplicate entry '662413617133735936-1729057440-700500076079419392' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.794417] [YB420AC531B8-00060B7DAD2B73E0-0-0] 2024-10-16 13:45:14.795 ERROR 4486 --- [metric-archive-async-10,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=2323956) Duplicate entry '620099296927748096-1729057440-620100047733329920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.794917] [YB420AC531B8-00060B7DB46FEB03-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '620099296927748096-1729057440-620100047733329920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.794917] [YB420AC531B8-00060B7DB46FEB03-0-0] 2024-10-16 13:45:14.798 ERROR 4486 --- [metric-archive-async-4,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=3866725) Duplicate entry '587105645062852608-1729057440-587106147037155328' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.797436] [YB420AC531B8-00060B7DB03B2550-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '587105645062852608-1729057440-587106147037155328' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.797436] [YB420AC531B8-00060B7DB03B2550-0-0] 2024-10-16 13:45:14.799 INFO 4486 --- [metric-archive-async-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057440, nodeCount=4, validCount=4 2024-10-16 13:45:14.799 ERROR 4486 --- [metric-archive-async-13,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=70525) Duplicate entry '587105645381619712-1729057440-587106147037155328' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.798977] [YB420AC531B8-00060B7D84EAB092-0-0]; nested exception is java.sql.BatchUpdateException: (conn=70525) Duplicate entry '587105645381619712-1729057440-587106147037155328' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.798977] [YB420AC531B8-00060B7D84EAB092-0-0] 2024-10-16 13:45:14.802 ERROR 4486 --- [metric-archive-async-3,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=2457395) Duplicate entry '657310995552210944-1729057440-663421513132216320' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.801494] [YB420AC531B8-00060B7DB41963BA-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2457395) Duplicate entry '657310995552210944-1729057440-663421513132216320' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.801494] [YB420AC531B8-00060B7DB41963BA-0-0] 2024-10-16 13:45:14.871 ERROR 4486 --- [metric-archive-async-6,,] c.o.o.m.storage.AsyncComputeDataDao : Archive node list to table metric_data_async error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_async` (`record_id`, `timestamp`, `data`, `interval`,`measurement_id`) VALUES (?, ?, ?, ?, ?)]; (conn=1525855) Duplicate entry '587075446682226688-1729057440-587075696121679872' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.871097] [YB420AC531B8-00060B7DB7688583-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1525855) Duplicate entry '587075446682226688-1729057440-587075696121679872' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:45:14.871097] [YB420AC531B8-00060B7DB7688583-0-0] 2024-10-16 13:45:14.908 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057440, nodeCount=136, affectRows=136 2024-10-16 13:45:14.919 INFO 4486 --- [metric-archive-async-10,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057440, nodeCount=147, affectRows=147 2024-10-16 13:45:14.930 INFO 4486 --- [metric-archive-async-21,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057440, nodeCount=168, affectRows=168 2024-10-16 13:45:14.945 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057440, nodeCount=151, affectRows=151 2024-10-16 13:45:14.953 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057440, nodeCount=175, affectRows=175 2024-10-16 13:45:15.023 INFO 4486 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057440, nodeCount=179, affectRows=179 2024-10-16 13:45:15.114 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057440, nodeCount=135, validCount=135 2024-10-16 13:45:15.166 INFO 4486 --- [metric-archive-async-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057440, nodeCount=147, validCount=147 2024-10-16 13:45:15.245 INFO 4486 --- [metric-archive-async-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057440, nodeCount=148, validCount=148 2024-10-16 13:45:15.252 INFO 4486 --- [metric-archive-async-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057440, nodeCount=166, validCount=166 2024-10-16 13:45:15.263 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057440, nodeCount=152, validCount=152 2024-10-16 13:45:15.264 INFO 4486 --- [ocp-async-9,7c5e7194f0892007,a3edbaa2b05811dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:15.293 INFO 4486 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057440, nodeCount=115, validCount=115 2024-10-16 13:45:16.031 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:16.888 INFO 4486 --- [ocp-schedules-16,c902a6dc730fd1a2,d3973d228c03a9e8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:45:16.888 INFO 4486 --- [ocp-schedules-16,b0b8958350a52ad9,db6f7ce0c110f781] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:45:16.890 INFO 4486 --- [ocp-schedules-16,b0b8958350a52ad9,db6f7ce0c110f781] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:45:16.919 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:16.920 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:16.932 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:16.933 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:16.960 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:16.961 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:16.962 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:16.963 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:16.963 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:45:16.994 INFO 4486 --- [prometheus-minute-schedule-1,aa0cff79332bc2f6,3d7c26c153f20776] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:16.999 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:17.000 INFO 4486 --- [prometheus-minute-schedule-1,aa0cff79332bc2f6,3d7c26c153f20776] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:17.000 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.001 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:17.002 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:45:17.002 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.006 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:45:17.007 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:17.020 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.020 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:17.032 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.044 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:17.045 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.045 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:17.046 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:45:17.051 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:45:17.055 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:17.057 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:17.069 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:17.070 INFO 4486 --- [ocp-schedules-2,282269bd89165e8e,0322200239d53043] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:45:17.070 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.071 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:45:17.082 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:45:17.090 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:45:17.091 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:45:17.121 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:17.123 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.123 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:17.127 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.131 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:45:17.134 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:17.162 INFO 4486 --- [ocp-schedules-9,48e2c2f62dff549f,55f238e4de729b03] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:45:17.162 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.162 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:17.163 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:45:17.172 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:17.173 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.173 INFO 4486 --- [ocp-schedules-25,1787c9edee704848,43cf4ff612c61774] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:45:17.174 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:45:17.183 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:45:17.183 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:45:17.188 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.188 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:17.214 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.214 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:17.240 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.240 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:17.264 INFO 4486 --- [ocp-async-5,26da850da99bb65b,bd0425cdbb0544b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:17.268 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.268 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:17.294 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:17.319 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:17.320 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.321 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:17.326 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:45:17.331 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:45:17.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:45:17.338 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:45:17.362 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:45:17.511 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:45:17.544 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:17.547 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:17.549 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:45:17.581 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:45:17.581 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:45:17.595 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,119338925c52407f,b3ea1b82beac37a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=119338925c52407f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:17.596 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,119338925c52407f,b3ea1b82beac37a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=119338925c52407f, duration=3 ms] 2024-10-16 13:45:19.024 INFO 4486 --- [ocp-schedules-13,0809402a00a5410d,60fd877ad64ac72e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:45:19.032 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:19.264 INFO 4486 --- [ocp-async-6,3314a382a034afde,bff35b4201ca64e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:19.758 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a4c7fe09cab6007d,23d8b7c3cc37854b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a4c7fe09cab6007d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:19.758 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a4c7fe09cab6007d,23d8b7c3cc37854b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a4c7fe09cab6007d, duration=1 ms] 2024-10-16 13:45:19.994 INFO 4486 --- [prometheus-minute-schedule-1,a97b6c0f3ae3616e,1df3285de49b6f49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:20.000 INFO 4486 --- [prometheus-minute-schedule-1,a97b6c0f3ae3616e,1df3285de49b6f49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:20.721 INFO 4486 --- [ocp-schedules-30,5d2a439ce7209ebc,b4c3bd8542eb10e6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:45:20.996 INFO 4486 --- [prometheus-cache-monitor1,96d1a7430176cc0b,f53ea7f0d1b8bf12] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13802,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:45:20.997 INFO 4486 --- [prometheus-cache-monitor1,96d1a7430176cc0b,f53ea7f0d1b8bf12] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:45:21.264 INFO 4486 --- [ocp-async-3,ef0a98c2909e9fe8,48579c09097526c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:21.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a762243517f6ee92,4d2c3ccc09288adb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a762243517f6ee92, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:21.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a762243517f6ee92,4d2c3ccc09288adb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a762243517f6ee92, duration=1 ms] 2024-10-16 13:45:21.655 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7143e238762540b7,868befdfcdcaf83d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7143e238762540b7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:21.655 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7143e238762540b7,868befdfcdcaf83d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7143e238762540b7, duration=1 ms] 2024-10-16 13:45:22.033 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:22.994 INFO 4486 --- [prometheus-minute-schedule-1,df0a98e39989f434,600488300175eb5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:22.999 INFO 4486 --- [prometheus-minute-schedule-1,df0a98e39989f434,600488300175eb5a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:23.005 INFO 4486 --- [ocp-async-4,a1f55c164ca8895d,dd26c3f4711973fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:45:23.166 INFO 4486 --- [ocp-schedules-11,e5bfac017c847103,5f8ca3b3212468d9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:45:23.203 INFO 4486 --- [ocp-schedules-8,35631e35dfa75dad,303e7f39aa5f60a7] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:45:23.229 WARN 4486 --- [scanner-13280128-1,e5bfac017c847103,1f3f006623734089] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:45:23.266 INFO 4486 --- [ocp-async-10,875e9b7ec35cb8e8,587df83abe1f1d22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:24.667 INFO 4486 --- [ocp-schedules-3,7a8c363d7d9517e4,b743961ff71bc57a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:45:25.034 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:25.265 INFO 4486 --- [ocp-async-10,fe32cac3847a41c5,375be86d4a01dc44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:25.450 INFO 4486 --- [ocp-schedules-24,75f817ecd52c3fb3,b2c2aef7888dab75] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:45:25.450 INFO 4486 --- [ocp-schedules-24,75f817ecd52c3fb3,b2c2aef7888dab75] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:45:25.479 INFO 4486 --- [Collect all tenants major compactions3,75f817ecd52c3fb3,5d08d5c49a64f24f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:45:25.480 INFO 4486 --- [Collect all tenants major compactions5,75f817ecd52c3fb3,0457f66be217e2b4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:45:25.480 INFO 4486 --- [Collect all tenants major compactions2,75f817ecd52c3fb3,978df3d52c52c13f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:45:25.480 INFO 4486 --- [Collect all tenants major compactions4,75f817ecd52c3fb3,1f185e10e2591e9f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:45:25.480 INFO 4486 --- [Collect all tenants major compactions1,75f817ecd52c3fb3,56c361e77383d11f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:45:25.491 INFO 4486 --- [Collect all tenants major compactions2,75f817ecd52c3fb3,978df3d52c52c13f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:25.492 INFO 4486 --- [Collect all tenants major compactions2,75f817ecd52c3fb3,978df3d52c52c13f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:45:25.494 INFO 4486 --- [Collect all tenants major compactions1,75f817ecd52c3fb3,56c361e77383d11f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:25.495 INFO 4486 --- [Collect all tenants major compactions3,75f817ecd52c3fb3,5d08d5c49a64f24f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:25.496 INFO 4486 --- [Collect all tenants major compactions1,75f817ecd52c3fb3,56c361e77383d11f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:45:25.496 INFO 4486 --- [Collect all tenants major compactions2,75f817ecd52c3fb3,978df3d52c52c13f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:45:25.496 INFO 4486 --- [Collect all tenants major compactions2,75f817ecd52c3fb3,978df3d52c52c13f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:45:25.497 INFO 4486 --- [Collect all tenants major compactions3,75f817ecd52c3fb3,5d08d5c49a64f24f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:45:25.498 INFO 4486 --- [Collect all tenants major compactions1,75f817ecd52c3fb3,56c361e77383d11f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:45:25.498 INFO 4486 --- [Collect all tenants major compactions1,75f817ecd52c3fb3,56c361e77383d11f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:45:25.501 INFO 4486 --- [Collect all tenants major compactions3,75f817ecd52c3fb3,5d08d5c49a64f24f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:45:25.501 INFO 4486 --- [Collect all tenants major compactions3,75f817ecd52c3fb3,5d08d5c49a64f24f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:45:25.508 INFO 4486 --- [Collect all tenants major compactions5,75f817ecd52c3fb3,0457f66be217e2b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:25.509 INFO 4486 --- [Collect all tenants major compactions4,75f817ecd52c3fb3,1f185e10e2591e9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:25.510 INFO 4486 --- [Collect all tenants major compactions5,75f817ecd52c3fb3,0457f66be217e2b4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:45:25.511 INFO 4486 --- [Collect all tenants major compactions4,75f817ecd52c3fb3,1f185e10e2591e9f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:45:25.514 INFO 4486 --- [Collect all tenants major compactions4,75f817ecd52c3fb3,1f185e10e2591e9f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:45:25.514 INFO 4486 --- [Collect all tenants major compactions4,75f817ecd52c3fb3,1f185e10e2591e9f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:45:25.515 INFO 4486 --- [Collect all tenants major compactions5,75f817ecd52c3fb3,0457f66be217e2b4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:45:25.515 INFO 4486 --- [Collect all tenants major compactions5,75f817ecd52c3fb3,0457f66be217e2b4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:45:25.516 INFO 4486 --- [ocp-schedules-24,75f817ecd52c3fb3,b2c2aef7888dab75] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:45:25.848 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4fac191feeff8297,a001729aac9a6555] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=4fac191feeff8297, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:45:25.849 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4fac191feeff8297,a001729aac9a6555] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=4fac191feeff8297, duration=2 ms] 2024-10-16 13:45:25.994 INFO 4486 --- [prometheus-minute-schedule-1,cdaccdcf6cde2753,460656dc8d1396d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:26.000 INFO 4486 --- [prometheus-minute-schedule-1,cdaccdcf6cde2753,460656dc8d1396d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:26.152 INFO 4486 --- [ocp-schedules-15,283c853a4a590d41,2333865f38665a85] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:45:27.265 INFO 4486 --- [ocp-async-8,84feeac2414f738a,7c0b56ef0886d66f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:27.312 INFO 4486 --- [ocp-schedules-11,7fb5521cb685f186,3d7a7092171811e1] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:45:27.311+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:45:27.313 INFO 4486 --- [ocp-schedules-11,7fb5521cb685f186,3d7a7092171811e1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:45:27.314 INFO 4486 --- [ocp-schedules-11,7fb5521cb685f186,3d7a7092171811e1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:45:27.315 INFO 4486 --- [ocp-schedules-11,7fb5521cb685f186,3d7a7092171811e1] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:45:27.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a9a8ef4de4268cf4,6631322ea487781b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a9a8ef4de4268cf4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:27.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a9a8ef4de4268cf4,6631322ea487781b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a9a8ef4de4268cf4, duration=1 ms] 2024-10-16 13:45:28.035 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:28.994 INFO 4486 --- [prometheus-minute-schedule-1,881d55d8ad481920,25a6ff5ffeb60f10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:29.000 INFO 4486 --- [prometheus-minute-schedule-1,881d55d8ad481920,25a6ff5ffeb60f10] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:29.265 INFO 4486 --- [ocp-async-9,0a7d4c457818cbd7,3a3af3c5f04388ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:29.764 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f12b276e1e90c869,81b7a05701d7ccf2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f12b276e1e90c869, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:29.764 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f12b276e1e90c869,81b7a05701d7ccf2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f12b276e1e90c869, duration=2 ms] 2024-10-16 13:45:30.102 INFO 4486 --- [prometheus-second-schedule-1,5cf459baf49970a9,68f16d45eef9ab7c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:45:30.728 INFO 4486 --- [ocp-schedules-27,5470bf3ae5029ac0,24021e94a3d19584] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:45:31.037 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:31.266 INFO 4486 --- [ocp-async-7,b94cfe99490fb6a6,245689f7535f6b41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:31.378 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,289496c6625b196c,7b14da008c11be88] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=289496c6625b196c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:31.378 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,289496c6625b196c,7b14da008c11be88] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=289496c6625b196c, duration=2 ms] 2024-10-16 13:45:31.661 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,33a9f535f7addfe8,ec630ca583ffdf78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=33a9f535f7addfe8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:31.661 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,33a9f535f7addfe8,ec630ca583ffdf78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=33a9f535f7addfe8, duration=1 ms] 2024-10-16 13:45:31.994 INFO 4486 --- [prometheus-minute-schedule-1,3dc43dfbe84a6f65,94ada4e12d4d2483] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:32.000 INFO 4486 --- [prometheus-minute-schedule-1,3dc43dfbe84a6f65,94ada4e12d4d2483] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:33.005 INFO 4486 --- [ocp-async-2,6ab516a50655a7d4,6f187c86bf30dca8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:45:33.266 INFO 4486 --- [ocp-async-10,beca19bea14e88a3,939ccd9040c2dc56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:33.516 INFO 4486 --- [ocp-schedules-20,329aee7adecb563f,c30d4ca381e3207b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:45:33.546 WARN 4486 --- [scanner-13280424-1,329aee7adecb563f,42fec52b337f17ae] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:45:33.607 INFO 4486 --- [alarm-detect-5,329aee7adecb563f,d5c001e18bab2a1d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057500 ~ 1729057523 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057515, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057515, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057515=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:45:34.037 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:34.994 INFO 4486 --- [prometheus-minute-schedule-1,8e8ed7a13c750164,4900c698f1a56f2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:35.000 INFO 4486 --- [prometheus-minute-schedule-1,8e8ed7a13c750164,4900c698f1a56f2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:35.266 INFO 4486 --- [ocp-async-8,80797ba0203c0e5b,3fcb21e75e6e4053] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:36.022 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9fffc3268d2e8464,eac31e608b4fe1fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=9fffc3268d2e8464, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:45:36.023 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9fffc3268d2e8464,eac31e608b4fe1fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=9fffc3268d2e8464, duration=2 ms] 2024-10-16 13:45:37.038 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:37.063 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:45:37.064 INFO 4486 --- [ocp-schedules-23,b62bb6d56e66e377,8afd2150c37a030b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:45:37.066 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:45:37.067 INFO 4486 --- [ocp-schedules-24,7c6f11a726754f15,81a709ced6df2f9a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:45:37.069 INFO 4486 --- [ocp-schedules-24,7c6f11a726754f15,81a709ced6df2f9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:45:37.069 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:45:37.069 INFO 4486 --- [ocp-schedules-23,b62bb6d56e66e377,8afd2150c37a030b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:45:37.081 INFO 4486 --- [ocp-schedules-5,bbd6a2a687643909,4e082a352a21c4ad] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:45:37.083 INFO 4486 --- [ocp-schedules-5,bbd6a2a687643909,4e082a352a21c4ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:45:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:45:37.091 INFO 4486 --- [ocp-schedules-21,b36df025ad4b78e7,2b693226c3a9c757] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:45:37.091 INFO 4486 --- [ocp-schedules-3,1b4f9eadce3ed939,fd69fcbd049e5e3f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:45:37.091 INFO 4486 --- [ocp-schedules-9,c6c983263c639f29,ed01d0d312792fb5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:45:37.092 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:45:37.093 INFO 4486 --- [host-check-agent-status1,c6c983263c639f29,01e1ac42e1848760] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:45:37.093 INFO 4486 --- [host-check-agent-status2,c6c983263c639f29,f8f40f2bf83577f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:45:37.093 INFO 4486 --- [host-check-clockdiff1,1b4f9eadce3ed939,b5fac6f1bb52b8a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:45:37.093 INFO 4486 --- [host-check-clockdiff2,1b4f9eadce3ed939,4419e02da06713ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:45:37.095 INFO 4486 --- [host-check-clockdiff4,1b4f9eadce3ed939,8a7cc3eab3927e36] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:45:37.095 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:45:37.095 INFO 4486 --- [host-check-clockdiff3,1b4f9eadce3ed939,d26f127a91c97d86] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:45:37.095 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:45:37.095 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:45:37.096 INFO 4486 --- [host-check-agent-status3,c6c983263c639f29,ce9529584d03dd46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:45:37.096 INFO 4486 --- [host-check-agent-status4,c6c983263c639f29,e6722b9614f4f362] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:45:37.102 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:37.102 INFO 4486 --- [host-check-agent-status2,c6c983263c639f29,f8f40f2bf83577f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:45:37.103 INFO 4486 --- [host-check-agent-status1,c6c983263c639f29,01e1ac42e1848760] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:37.104 INFO 4486 --- [host-check-agent-status4,c6c983263c639f29,e6722b9614f4f362] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:45:37.105 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:45:37.105 INFO 4486 --- [host-check-agent-status3,c6c983263c639f29,ce9529584d03dd46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:45:37.106 INFO 4486 --- [host-check-agent-status2,c6c983263c639f29,f8f40f2bf83577f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:45:37.106 INFO 4486 --- [host-check-agent-status4,c6c983263c639f29,e6722b9614f4f362] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:45:37.106 INFO 4486 --- [host-check-agent-status1,c6c983263c639f29,01e1ac42e1848760] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:37.106 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:45:37.106 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:37.107 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:45:37.107 INFO 4486 --- [host-check-agent-status3,c6c983263c639f29,ce9529584d03dd46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:45:37.142 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:45:37.135 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:45:37.152 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:45:37.153 INFO 4486 --- [host-check-clockdiff3,1b4f9eadce3ed939,d26f127a91c97d86] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057537 4 4 , error= 2024-10-16 13:45:37.159 INFO 4486 --- [host-check-clockdiff4,1b4f9eadce3ed939,8a7cc3eab3927e36] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057537 3 3 , error= 2024-10-16 13:45:37.163 INFO 4486 --- [host-check-clockdiff1,1b4f9eadce3ed939,b5fac6f1bb52b8a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057537 0 0 , error= 2024-10-16 13:45:37.167 INFO 4486 --- [host-check-clockdiff2,1b4f9eadce3ed939,4419e02da06713ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057537 4 4 , error= 2024-10-16 13:45:37.170 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.171 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:37.172 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:45:37.187 INFO 4486 --- [host-check-agent-status4,c6c983263c639f29,e6722b9614f4f362] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:45:37.187 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:45:37.189 INFO 4486 --- [host-check-agent-status3,c6c983263c639f29,ce9529584d03dd46] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:45:37.189 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:45:37.189204018+08:00 2024-10-16 13:45:37.190 INFO 4486 --- [host-check-agent-status1,c6c983263c639f29,01e1ac42e1848760] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:45:37.193 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:45:37.190 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:45:37.189+08:00 2024-10-16 13:45:37.194 INFO 4486 --- [host-check-agent-status4,c6c983263c639f29,e6722b9614f4f362] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:45:37.194 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:45:37.194 INFO 4486 --- [host-check-agent-status4,c6c983263c639f29,e6722b9614f4f362] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:45:37.194 INFO 4486 --- [host-check-agent-status4,c6c983263c639f29,e6722b9614f4f362] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:45:37.195 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:45:37.195 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:45:37.198549437+08:00 2024-10-16 13:45:37.196 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:45:37.195+08:00 2024-10-16 13:45:37.196 INFO 4486 --- [host-check-agent-status3,c6c983263c639f29,ce9529584d03dd46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:45:37.196 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:45:37.189+08:00 and status AVAILABLE 2024-10-16 13:45:37.197 INFO 4486 --- [host-check-agent-status3,c6c983263c639f29,ce9529584d03dd46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:45:37.197 INFO 4486 --- [host-check-agent-status3,c6c983263c639f29,ce9529584d03dd46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:45:37.197 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.197 INFO 4486 --- [host-check-agent-status1,c6c983263c639f29,01e1ac42e1848760] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:45:37.197 INFO 4486 --- [host-check-agent-status1,c6c983263c639f29,01e1ac42e1848760] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:45:37.197 INFO 4486 --- [host-check-agent-status1,c6c983263c639f29,01e1ac42e1848760] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:45:37.197 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:45:37.199909351+08:00 2024-10-16 13:45:37.197 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:45:37.200588221+08:00 2024-10-16 13:45:37.198 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:45:37.197+08:00 2024-10-16 13:45:37.198 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:37.198 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:45:37.197+08:00 2024-10-16 13:45:37.198 INFO 4486 --- [host-check-agent-status2,c6c983263c639f29,f8f40f2bf83577f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:45:37.199 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:45:37.199 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:45:37.195+08:00 and status AVAILABLE 2024-10-16 13:45:37.201 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:45:37.197+08:00 and status AVAILABLE 2024-10-16 13:45:37.201 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:45:37.197+08:00 and status AVAILABLE 2024-10-16 13:45:37.201 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:45:37.202 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:45:37.203 INFO 4486 --- [host-check-agent-status2,c6c983263c639f29,f8f40f2bf83577f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:45:37.203 INFO 4486 --- [host-check-agent-status2,c6c983263c639f29,f8f40f2bf83577f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:45:37.203 INFO 4486 --- [host-check-agent-status2,c6c983263c639f29,f8f40f2bf83577f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:45:37.203 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:45:37.203 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:45:37.206 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.207 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:37.209 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:37.211 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:45:37.211 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:45:37.212 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:45:37.212 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:45:37.213 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:45:37.214 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:45:37.214 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:45:37.216 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.218 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.219 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.220 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:45:37.222 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:45:37.230 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.231 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:45:37.234 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.234 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.234 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.234 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.234 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.235 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.235 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:37.236 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:37.237 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:37.240 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:45:37.242 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.243 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:45:37.252 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.253 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:45:37.255 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:45:37.255 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.256 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:45:37.266 INFO 4486 --- [ocp-async-7,d0f8e99360a62eb2,313c9ff553bcac3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:37.267 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.268 INFO 4486 --- [ocp-schedules-25,e701f5cee9b5714d,ab73e922b50230fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:45:37.270 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:45:37.276 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:45:37.280 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:45:37.284 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:45:37.290 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:45:37.291 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:45:37.294 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:45:37.304 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:37.305 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:45:37.309 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:37.313 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:37.320 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:45:37.336 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:45:37.346 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:45:37.348 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:45:37.359 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:45:37.361 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:45:37.364 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:37.372 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:45:37.374 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:45:37.375 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:37.385 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:45:37.388 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:37.397 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:45:37.405 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:45:37.410 INFO 4486 --- [host-check-host-status3,b36df025ad4b78e7,5ec272e99bb21092] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:45:37.410 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:45:37.420 INFO 4486 --- [host-check-host-status4,b36df025ad4b78e7,4ee09ec06faaddbf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:45:37.431 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:45:37.445 INFO 4486 --- [host-check-host-status2,b36df025ad4b78e7,a8ee2d8d1b139775] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:45:37.463 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:45:37.478 INFO 4486 --- [host-check-host-status1,b36df025ad4b78e7,bebd4dec3e3cd170] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:45:37.485 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:37.488 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.489 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:45:37.491 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.491 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.492 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:45:37.506 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.506 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.506 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.506 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.506 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.507 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.534 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:45:37.572 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:45:37.607 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b387c1f6669f6373,2991a27fb827c58f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b387c1f6669f6373, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:37.607 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b387c1f6669f6373,2991a27fb827c58f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b387c1f6669f6373, duration=1 ms] 2024-10-16 13:45:37.619 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:37.636 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:37.637 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.638 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:45:37.639 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:45:37.641 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.642 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:45:37.643 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.643 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.644 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:45:37.655 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.655 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.655 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.655 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.655 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.656 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.663 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:45:37.856 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:45:37.902 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:37.904 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.905 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:45:37.907 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.909 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.910 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:45:37.919 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.919 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.919 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.919 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.919 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:37.919 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:37.937 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:45:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:45:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:45:37.994 INFO 4486 --- [prometheus-minute-schedule-1,9dbd56dd0be8a3b4,fcea15d9de2746a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:38.000 INFO 4486 --- [prometheus-minute-schedule-1,9dbd56dd0be8a3b4,fcea15d9de2746a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:38.148 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:45:38.196 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:38.197 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:38.198 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:45:38.199 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:38.200 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:38.201 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:45:38.216 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:38.216 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:38.216 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:38.216 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:38.216 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:45:38.217 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:38.287 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:45:38.471 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:45:38.292006, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:45:38.471 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:45:38.292006, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:45:38.472 INFO 4486 --- [ocp-schedules-11,307c084caf7e3d31,779a47d1c32c8c68] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13802,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:45:38.472 INFO 4486 --- [ocp-schedules-11,307c084caf7e3d31,779a47d1c32c8c68] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:45:38.506 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:45:38.292006} } 2024-10-16 13:45:38.563 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:45:38.570 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:45:38.573 INFO 4486 --- [ocp-schedules-28,896e7ffcc3a8e09c,41397ec450961b4b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:45:39.030 INFO 4486 --- [ocp-schedules-21,46d4d17976548eda,e325f8710639c7b0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:45:39.267 INFO 4486 --- [ocp-async-10,4d1a87df0020528d,07b2e705b90f0d1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:39.769 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,aa80e0ff3b099be7,4bfe3e51cc65a977] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=aa80e0ff3b099be7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:39.770 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,aa80e0ff3b099be7,4bfe3e51cc65a977] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=aa80e0ff3b099be7, duration=2 ms] 2024-10-16 13:45:40.006 INFO 4486 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:45:40.007 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:45:40.027 INFO 4486 --- [Check obproxy1,3ab8d4bf087f70dc,c1de3a9e4662d746] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:45:40.028 INFO 4486 --- [Check obproxy1,3ab8d4bf087f70dc,c1de3a9e4662d746] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:40.031 INFO 4486 --- [Check obproxy2,8dfd99710cdfe14e,bf69c771ed568014] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:45:40.032 INFO 4486 --- [Check obproxy2,8dfd99710cdfe14e,bf69c771ed568014] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:40.033 INFO 4486 --- [Collect obproxy2,221e0f8f90b6c451,b12bfbb79bd749c7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:45:40.033 INFO 4486 --- [Collect obproxy2,221e0f8f90b6c451,b12bfbb79bd749c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:40.034 INFO 4486 --- [Collect obproxy1,f741dd1f93cbc8d2,a886aee86ff2aa80] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:45:40.034 INFO 4486 --- [Collect obproxy1,f741dd1f93cbc8d2,a886aee86ff2aa80] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:45:40.040 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:40.053 INFO 4486 --- [Check obproxy1,3ab8d4bf087f70dc,c1de3a9e4662d746] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:45:40.053 INFO 4486 --- [Check obproxy1,3ab8d4bf087f70dc,c1de3a9e4662d746] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:45:40.053 INFO 4486 --- [Collect obproxy1,f741dd1f93cbc8d2,a886aee86ff2aa80] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:45:40.055 INFO 4486 --- [Collect obproxy2,221e0f8f90b6c451,b12bfbb79bd749c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:45:40.057 INFO 4486 --- [Collect obproxy1,f741dd1f93cbc8d2,a886aee86ff2aa80] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:45:40.057 INFO 4486 --- [Check obproxy2,8dfd99710cdfe14e,bf69c771ed568014] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:45:40.057 INFO 4486 --- [Check obproxy2,8dfd99710cdfe14e,bf69c771ed568014] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:45:40.061 INFO 4486 --- [Collect obproxy1,f741dd1f93cbc8d2,a886aee86ff2aa80] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:45:40.062 INFO 4486 --- [Collect obproxy2,221e0f8f90b6c451,b12bfbb79bd749c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:45:40.068 INFO 4486 --- [Collect obproxy2,221e0f8f90b6c451,b12bfbb79bd749c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:45:40.733 INFO 4486 --- [ocp-schedules-12,7af99cf8f3515907,4bbf7022d1f214ab] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:45:40.994 INFO 4486 --- [prometheus-minute-schedule-1,6f29557bd2705521,e74f56fa2be7d70f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:40.995 INFO 4486 --- [prometheus-hour-schedule-1,4374aa230c3a9d1c,a2fb147b31f78928] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:45:41.002 INFO 4486 --- [prometheus-minute-schedule-1,6f29557bd2705521,e74f56fa2be7d70f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:41.267 INFO 4486 --- [ocp-async-1,ed58791ff57a56e9,62969db99b7faa90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:41.383 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,8f278fd37f3dbf16,384017a6b3e57e46] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8f278fd37f3dbf16, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:41.384 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,8f278fd37f3dbf16,384017a6b3e57e46] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8f278fd37f3dbf16, duration=1 ms] 2024-10-16 13:45:41.666 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6c6f786da1ca1a35,27504a674e3f42d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6c6f786da1ca1a35, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:41.666 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6c6f786da1ca1a35,27504a674e3f42d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6c6f786da1ca1a35, duration=0 ms] 2024-10-16 13:45:42.124 INFO 4486 --- [ocp-schedules-26,3a9acf9d8ca38559,80c8bc08ccf37303] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:45:43.005 INFO 4486 --- [ocp-async-1,4255b015673570de,bcce1f3bb1d0e649] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:45:43.041 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:43.267 INFO 4486 --- [ocp-async-2,24a5cdd25d04b98f,31452e7fc7725491] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:43.691 INFO 4486 --- [ocp-schedules-12,0c734e2636fadf85,ad3e542e8e6e8ea6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:45:43.729 WARN 4486 --- [scanner-13280683-1,0c734e2636fadf85,4bbde223546c3903] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:45:43.797 INFO 4486 --- [alarm-detect-1,0c734e2636fadf85,4deb66b7aeda8ddc] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057515 ~ 1729057533 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057530, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057530, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057530=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:45:43.994 INFO 4486 --- [prometheus-minute-schedule-1,e51c585e891c42eb,58ca71bd456e855f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:44.000 INFO 4486 --- [prometheus-minute-schedule-1,e51c585e891c42eb,58ca71bd456e855f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:45.267 INFO 4486 --- [ocp-async-4,42ec835601bb2554,039f2cd054789ead] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:45.510 INFO 4486 --- [ocp-schedules-3,9218bba588724f97,f0dd0e830638eba6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:45:45.515 INFO 4486 --- [ocp-schedules-3,9218bba588724f97,f0dd0e830638eba6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:45:45.515 INFO 4486 --- [ocp-schedules-3,9218bba588724f97,f0dd0e830638eba6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:45:45.530 INFO 4486 --- [ocp-schedules-25,4ee10224c82b5d42,303f53f3ed4922b7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:45:45.530 INFO 4486 --- [ocp-schedules-25,4ee10224c82b5d42,303f53f3ed4922b7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:45:46.042 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:46.994 INFO 4486 --- [prometheus-minute-schedule-1,ca764a0d34c38686,b2c1644265406123] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:47.000 INFO 4486 --- [prometheus-minute-schedule-1,ca764a0d34c38686,b2c1644265406123] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:47.071 INFO 4486 --- [ocp-schedules-24,7c6f11a726754f15,81a709ced6df2f9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:45:47.167 INFO 4486 --- [ocp-schedules-7,eabaa6d5323abc4a,0bdffe3f79313470] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:45:47.202 INFO 4486 --- [ocp-schedules-5,bbd6a2a687643909,4e082a352a21c4ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:45:47.268 INFO 4486 --- [ocp-async-8,7dd8b612ecc41f0a,1e55c0b0c6b1a70d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:47.581 INFO 4486 --- [ocp-schedules-16,b0b8958350a52ad9,db6f7ce0c110f781] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:45:47.582 INFO 4486 --- [ocp-schedules-16,66a6f06ea9273007,be2f5af3b71b70eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:45:47.584 INFO 4486 --- [ocp-schedules-16,66a6f06ea9273007,be2f5af3b71b70eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:45:47.609 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.610 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:47.616 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e83545995325d2f1,4493a49bbd98541b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e83545995325d2f1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:47.616 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e83545995325d2f1,4493a49bbd98541b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e83545995325d2f1, duration=2 ms] 2024-10-16 13:45:47.626 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.627 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:47.644 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:47.644 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.645 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.645 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:45:47.645 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:47.683 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:47.683 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.684 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:47.685 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:45:47.685 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.688 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:45:47.690 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:47.699 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:47.699 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:47.711 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:47.722 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:47.722 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.723 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:47.724 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:45:47.728 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:45:47.733 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:47.735 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:47.751 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:47.753 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.754 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:45:47.812 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:45:47.832 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:45:47.832 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:45:47.845 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:47.847 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.848 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:45:47.851 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.854 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:45:47.858 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:47.884 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:47.884 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:47.889 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:45:47.899 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:47.900 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:47.900 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:45:47.910 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:45:47.910 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:45:47.912 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:47.912 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:47.938 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:47.938 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:47.965 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:47.965 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:45:47.994 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:47.994 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:45:48.020 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:45:48.046 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:48.047 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:48.048 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:45:48.053 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:45:48.057 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:45:48.061 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:45:48.062 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:45:48.087 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:45:48.224 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:45:48.254 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:45:48.256 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:48.257 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:45:48.286 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:45:48.286 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:45:48.760 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:48.760 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:45:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:45:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:45:48.763 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:45:49.043 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:49.268 INFO 4486 --- [ocp-async-6,bd67fae40680299c,07ad34d0cf8dea6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:49.774 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d7c25a4f2a47fb11,35a6151f08ec6534] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d7c25a4f2a47fb11, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:49.774 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d7c25a4f2a47fb11,35a6151f08ec6534] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d7c25a4f2a47fb11, duration=1 ms] 2024-10-16 13:45:49.994 INFO 4486 --- [prometheus-minute-schedule-1,903ed2e335c4b663,c4a60662da950fa6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:49.999 INFO 4486 --- [prometheus-minute-schedule-1,903ed2e335c4b663,c4a60662da950fa6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:50.458 INFO 4486 --- [ocp-schedules-9,e5cc74b8916568e9,d9597fb8ac8903d1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:45:50.458 INFO 4486 --- [ocp-schedules-9,e5cc74b8916568e9,d9597fb8ac8903d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:45:50.459 INFO 4486 --- [ocp-schedules-9,e5cc74b8916568e9,d9597fb8ac8903d1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:45:50.459 INFO 4486 --- [ocp-schedules-9,e5cc74b8916568e9,d9597fb8ac8903d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:45:50.460 INFO 4486 --- [ocp-schedules-9,e5cc74b8916568e9,d9597fb8ac8903d1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:45:50.460 INFO 4486 --- [ocp-schedules-9,e5cc74b8916568e9,d9597fb8ac8903d1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:45:50.612 INFO 4486 --- [ocp-schedules-20,8e484cc210a8918c,ceff294a382acd0e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:45:50.613 INFO 4486 --- [ocp-schedules-20,8e484cc210a8918c,ceff294a382acd0e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:45:50.613 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:45:50.613 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:45:50.613 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:45:50.613 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:45:50.613 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:45:50.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:45:50.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:45:50.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:45:50.618 INFO 4486 --- [ocp-schedules-20,8e484cc210a8918c,ceff294a382acd0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:45:50.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:45:50.619 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:45:50.619 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:45:50.619 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:45:50.619 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:45:50.620 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:45:50.620 INFO 4486 --- [ocp-schedules-20,8e484cc210a8918c,ceff294a382acd0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:45:50.620 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:45:50.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:45:50.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:45:50.645 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:45:50.645 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:45:50.645 INFO 4486 --- [ocp-schedules-20,8e484cc210a8918c,ceff294a382acd0e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:45:50.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:45:50.656 INFO 4486 --- [ocp-schedules-20,8e484cc210a8918c,ceff294a382acd0e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:45:50.686 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:45:50.686 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:45:50.697 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:45:50.697 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:45:50.697 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:45:50.709 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:45:50.709 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:45:50.709 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:45:50.715 INFO 4486 --- [ocp-schedules-28,1d86e07fcee73537,6019fb970cab1f6e] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:45:50.738 INFO 4486 --- [ocp-schedules-19,906daa7d39fd3245,d461846f24cf561b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:45:51.268 INFO 4486 --- [ocp-async-3,f4b27733ecc0396a,05da148c4e53fae2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:51.388 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7251ac7c188dedb6,764ca7833e1e38b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7251ac7c188dedb6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:51.389 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7251ac7c188dedb6,764ca7833e1e38b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7251ac7c188dedb6, duration=2 ms] 2024-10-16 13:45:51.674 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6c01040cf40aeae1,9d43791e3504aa7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6c01040cf40aeae1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:51.675 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6c01040cf40aeae1,9d43791e3504aa7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6c01040cf40aeae1, duration=1 ms] 2024-10-16 13:45:52.045 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:52.816 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,433637eef5c44e9b,a14ddf950bfedc87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=433637eef5c44e9b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:45:52.816 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,433637eef5c44e9b,a14ddf950bfedc87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=433637eef5c44e9b, duration=2 ms] 2024-10-16 13:45:52.994 INFO 4486 --- [prometheus-minute-schedule-1,af17ed32a430e7fb,21719752a1e4918c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:53.000 INFO 4486 --- [prometheus-minute-schedule-1,af17ed32a430e7fb,21719752a1e4918c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:53.005 INFO 4486 --- [ocp-async-4,1d45f44c6f412456,8415016dfbd55165] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:45:53.268 INFO 4486 --- [ocp-async-8,18ed9cf8f32b498d,e71ed5e7069f651b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:53.875 INFO 4486 --- [ocp-schedules-14,2580261f822642e0,00e28b235a78642f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:45:53.959 WARN 4486 --- [scanner-13281052-1,2580261f822642e0,37a4417b2bbc6bbd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:45:55.046 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:55.268 INFO 4486 --- [ocp-async-2,978c1bd6a9ae4ef7,c99920ce40518b70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:55.994 INFO 4486 --- [prometheus-minute-schedule-1,09c1efc0320652cd,bc5e32c3a8b0339b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:56.000 INFO 4486 --- [prometheus-minute-schedule-1,09c1efc0320652cd,bc5e32c3a8b0339b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:57.269 INFO 4486 --- [ocp-async-3,81a0b74e2f154198,8283d752c03468a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:57.623 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,27f5c24dcd2ea9b7,77d0bd2a8fb25311] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=27f5c24dcd2ea9b7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:57.623 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,27f5c24dcd2ea9b7,77d0bd2a8fb25311] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=27f5c24dcd2ea9b7, duration=2 ms] 2024-10-16 13:45:58.047 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:45:58.994 INFO 4486 --- [prometheus-minute-schedule-1,065ed484c8a5e87a,ea05cb7acf307cfb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057560, interval=60 2024-10-16 13:45:59.002 INFO 4486 --- [prometheus-minute-schedule-1,065ed484c8a5e87a,ea05cb7acf307cfb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:45:59.035 INFO 4486 --- [ocp-schedules-25,f4638145b606c713,a07b3ec02587bea5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:45:59.269 INFO 4486 --- [ocp-async-1,1a2afd80c6509169,181328c619a057ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:45:59.778 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ecc25be44f2eabb6,073bca6e78e65dbe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ecc25be44f2eabb6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:45:59.778 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,ecc25be44f2eabb6,073bca6e78e65dbe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ecc25be44f2eabb6, duration=1 ms] 2024-10-16 13:46:00.744 INFO 4486 --- [ocp-schedules-28,793c819d282ab9bd,802cf830aae6a8ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:46:01.048 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:01.269 INFO 4486 --- [ocp-async-8,aba8e67ba16ba808,ae43257d687c2b65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:01.393 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0d56e482d8fe56e5,f1a60e272851d6b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0d56e482d8fe56e5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:01.393 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0d56e482d8fe56e5,f1a60e272851d6b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0d56e482d8fe56e5, duration=1 ms] 2024-10-16 13:46:01.679 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,40ebf1d8e1d8d041,143661848e40073b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=40ebf1d8e1d8d041, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:01.679 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,40ebf1d8e1d8d041,143661848e40073b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=40ebf1d8e1d8d041, duration=1 ms] 2024-10-16 13:46:01.827 INFO 4486 --- [ocp-schedules-7,04b7fac24f60fe2f,57dfe422c14081cc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:46:01.827+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:46:01.832 INFO 4486 --- [ocp-schedules-7,04b7fac24f60fe2f,57dfe422c14081cc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:46:01.836 INFO 4486 --- [ocp-schedules-7,04b7fac24f60fe2f,57dfe422c14081cc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:46:01.842 INFO 4486 --- [ocp-schedules-7,04b7fac24f60fe2f,57dfe422c14081cc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster glob_3zone for slowSqlStat, sql count is 0. 2024-10-16 13:46:01.994 INFO 4486 --- [prometheus-minute-schedule-1,0ac3ce25a2e8c974,6550f35600c4f97b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:02.001 INFO 4486 --- [prometheus-minute-schedule-1,0ac3ce25a2e8c974,6550f35600c4f97b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:03.005 INFO 4486 --- [ocp-async-6,505289607d95ff21,ccafc5b056dea853] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:46:03.269 INFO 4486 --- [ocp-async-7,5c7c028b201390e2,41776cf021b1dd07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:03.523 INFO 4486 --- [ocp-schedules-17,feb0ab94633f21eb,8c364a6201493502] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:46:03.525 INFO 4486 --- [ocp-schedules-17,feb0ab94633f21eb,8c364a6201493502] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:03.527 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:46:03.527 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:46:03.528 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:46:04.050 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:04.065 INFO 4486 --- [ocp-schedules-23,bb5f31ba96bc7773,d7c747ff4d2c5134] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:46:04.106 WARN 4486 --- [scanner-13281247-1,bb5f31ba96bc7773,2cadb2dff5a984bf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:46:04.194 INFO 4486 --- [alarm-detect-9,bb5f31ba96bc7773,85d89f5232d26493] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057530 ~ 1729057554 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057545, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057545, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057545=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:46:04.994 INFO 4486 --- [prometheus-minute-schedule-1,140d9ff87ad271d5,861c75c7af99f9d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:05.001 INFO 4486 --- [prometheus-minute-schedule-1,140d9ff87ad271d5,861c75c7af99f9d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:05.269 INFO 4486 --- [ocp-async-10,940d128e3c1d5704,a00f917a5e67f954] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:07.051 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:07.066 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:46:07.067 INFO 4486 --- [ocp-schedules-13,d87872fb50545328,95b093ed23a6ac8c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:46:07.069 INFO 4486 --- [ocp-schedules-13,d87872fb50545328,95b093ed23a6ac8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:07.081 INFO 4486 --- [ocp-schedules-19,2113b9e242a16763,2008f35f1026373e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:46:07.084 INFO 4486 --- [ocp-schedules-19,2113b9e242a16763,2008f35f1026373e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:07.086 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:46:07.089 INFO 4486 --- [ocp-schedules-25,cfebd7ac417611db,17a3d8efc3337279] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:46:07.090 INFO 4486 --- [ocp-schedules-6,4a56e0322cba971e,4a5e0baa42c5f6f8] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:46:07.090 INFO 4486 --- [host-check-agent-status1,cfebd7ac417611db,f7ccefb365e7cc3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:46:07.090 INFO 4486 --- [host-check-agent-status2,cfebd7ac417611db,271d15134f49714e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:46:07.091 INFO 4486 --- [host-check-agent-status3,cfebd7ac417611db,9f7ee45095799311] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:46:07.091 INFO 4486 --- [ocp-schedules-2,e055418316b795a7,5d9568bb33a0c48e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:46:07.091 INFO 4486 --- [host-check-clockdiff2,4a56e0322cba971e,7e3bb82d418dc4a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:46:07.091 INFO 4486 --- [host-check-clockdiff1,4a56e0322cba971e,feab38cbb1a0ae2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:46:07.092 INFO 4486 --- [host-check-clockdiff3,4a56e0322cba971e,6309457ed0e1c57b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:46:07.092 INFO 4486 --- [host-check-agent-status4,cfebd7ac417611db,e987b5f17327b03f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:46:07.093 INFO 4486 --- [host-check-clockdiff4,4a56e0322cba971e,2ceffaaa1f72581f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:46:07.093 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:46:07.094 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:46:07.094 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:46:07.094 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:46:07.100 INFO 4486 --- [host-check-agent-status2,cfebd7ac417611db,271d15134f49714e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:46:07.100 INFO 4486 --- [host-check-agent-status3,cfebd7ac417611db,9f7ee45095799311] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:46:07.101 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:07.106 INFO 4486 --- [host-check-agent-status4,cfebd7ac417611db,e987b5f17327b03f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:46:07.124 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:46:07.139 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:46:07.139 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:07.139 INFO 4486 --- [host-check-agent-status3,cfebd7ac417611db,9f7ee45095799311] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:46:07.139 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:46:07.140 INFO 4486 --- [host-check-clockdiff2,4a56e0322cba971e,7e3bb82d418dc4a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057567 4 4 , error= 2024-10-16 13:46:07.140 INFO 4486 --- [host-check-clockdiff3,4a56e0322cba971e,6309457ed0e1c57b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057567 4 4 , error= 2024-10-16 13:46:07.140 INFO 4486 --- [host-check-clockdiff1,4a56e0322cba971e,feab38cbb1a0ae2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057567 0 0 , error= 2024-10-16 13:46:07.139 INFO 4486 --- [host-check-agent-status2,cfebd7ac417611db,271d15134f49714e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:46:07.146 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:46:07.146 INFO 4486 --- [host-check-agent-status4,cfebd7ac417611db,e987b5f17327b03f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:46:07.143 INFO 4486 --- [host-check-clockdiff4,4a56e0322cba971e,2ceffaaa1f72581f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057567 3 3 , error= 2024-10-16 13:46:07.147 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:46:07.147 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:46:07.149 INFO 4486 --- [host-check-agent-status1,cfebd7ac417611db,f7ccefb365e7cc3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:07.149 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:07.151 INFO 4486 --- [host-check-agent-status1,cfebd7ac417611db,f7ccefb365e7cc3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:07.151 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:46:07.174 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:46:07.175 INFO 4486 --- [host-check-agent-status3,cfebd7ac417611db,9f7ee45095799311] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:46:07.175 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:46:07.177 INFO 4486 --- [host-check-agent-status2,cfebd7ac417611db,271d15134f49714e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:46:07.177 INFO 4486 --- [host-check-agent-status4,cfebd7ac417611db,e987b5f17327b03f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:46:07.177 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:46:07.178 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:46:07.178 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:46:07.177408238+08:00 2024-10-16 13:46:07.178 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.178 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:46:07.181457318+08:00 2024-10-16 13:46:07.179 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:46:07.182395153+08:00 2024-10-16 13:46:07.179 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:46:07.178+08:00 2024-10-16 13:46:07.179 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:07.179 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:46:07.178+08:00 2024-10-16 13:46:07.179 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:46:07.179+08:00 2024-10-16 13:46:07.180 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:46:07.183364872+08:00 2024-10-16 13:46:07.181 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:46:07.180+08:00 2024-10-16 13:46:07.181 INFO 4486 --- [host-check-agent-status3,cfebd7ac417611db,9f7ee45095799311] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:46:07.181 INFO 4486 --- [host-check-agent-status3,cfebd7ac417611db,9f7ee45095799311] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:46:07.181 INFO 4486 --- [host-check-agent-status3,cfebd7ac417611db,9f7ee45095799311] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:46:07.182 INFO 4486 --- [host-check-agent-status4,cfebd7ac417611db,e987b5f17327b03f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:46:07.182 INFO 4486 --- [host-check-agent-status4,cfebd7ac417611db,e987b5f17327b03f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:46:07.182 INFO 4486 --- [host-check-agent-status4,cfebd7ac417611db,e987b5f17327b03f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:46:07.182 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:46:07.178+08:00 and status AVAILABLE 2024-10-16 13:46:07.182 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:46:07.179+08:00 and status AVAILABLE 2024-10-16 13:46:07.182 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:46:07.178+08:00 and status AVAILABLE 2024-10-16 13:46:07.183 INFO 4486 --- [host-check-agent-status2,cfebd7ac417611db,271d15134f49714e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:46:07.183 INFO 4486 --- [host-check-agent-status2,cfebd7ac417611db,271d15134f49714e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:46:07.183 INFO 4486 --- [host-check-agent-status2,cfebd7ac417611db,271d15134f49714e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:46:07.183 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:46:07.180+08:00 and status AVAILABLE 2024-10-16 13:46:07.183 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:46:07.184 INFO 4486 --- [host-check-agent-status1,cfebd7ac417611db,f7ccefb365e7cc3a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:46:07.185 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:46:07.185 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:46:07.185 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:46:07.186 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:46:07.191 INFO 4486 --- [host-check-agent-status1,cfebd7ac417611db,f7ccefb365e7cc3a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:46:07.192 INFO 4486 --- [host-check-agent-status1,cfebd7ac417611db,f7ccefb365e7cc3a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:46:07.192 INFO 4486 --- [host-check-agent-status1,cfebd7ac417611db,f7ccefb365e7cc3a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:46:07.195 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:07.196 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:46:07.196 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:46:07.196 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:46:07.197 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:46:07.197 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:07.197 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:46:07.197 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:46:07.200 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.205 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:46:07.214 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.215 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:07.218 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:07.219 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:07.220 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:07.221 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:07.224 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.225 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:07.234 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.235 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:07.238 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.245 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:46:07.250 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:46:07.251 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:46:07.257 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:07.258 INFO 4486 --- [ocp-schedules-26,c8fea60b45bec26c,b8cb6d173d995e8c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:46:07.260 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:46:07.260 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:46:07.262 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:46:07.270 INFO 4486 --- [ocp-async-5,05483437ee11a9f2,00121b3e198d96a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:07.271 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:46:07.273 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:46:07.276 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:07.276 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:07.281 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:46:07.290 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:07.297 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:07.305 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:46:07.311 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:46:07.315 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:46:07.321 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:46:07.330 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:07.330 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:46:07.337 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:07.343 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:46:07.360 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:46:07.360 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:07.360 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:46:07.370 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:46:07.370 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:46:07.381 INFO 4486 --- [host-check-host-status4,e055418316b795a7,8fb594801f098bc5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:46:07.389 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:07.396 INFO 4486 --- [host-check-host-status3,e055418316b795a7,df46a5d951eefbdb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:46:07.397 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:46:07.408 INFO 4486 --- [host-check-host-status2,e055418316b795a7,6d86964c42be3449] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:46:07.442 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:46:07.452 INFO 4486 --- [host-check-host-status1,e055418316b795a7,56af0a925be89114] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:46:07.632 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,24368ec3f98cb661,615fde58b56d0406] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=24368ec3f98cb661, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:07.633 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,24368ec3f98cb661,615fde58b56d0406] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=24368ec3f98cb661, duration=2 ms] 2024-10-16 13:46:07.995 INFO 4486 --- [prometheus-minute-schedule-1,f12c11cff52f087c,de2d4faef90fa7a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:08.004 INFO 4486 --- [prometheus-minute-schedule-1,f12c11cff52f087c,de2d4faef90fa7a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:09.270 INFO 4486 --- [ocp-async-2,d505914a8dcad3ff,50a10a664a9e2cc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:09.615 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,95bbd59702fa48c9,95b9090f59b6bfca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=95bbd59702fa48c9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:46:09.616 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,95bbd59702fa48c9,95b9090f59b6bfca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=95bbd59702fa48c9, duration=2 ms] 2024-10-16 13:46:09.782 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f2076b97ac80099b,e5dd4932b34ff339] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f2076b97ac80099b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:09.783 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f2076b97ac80099b,e5dd4932b34ff339] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f2076b97ac80099b, duration=1 ms] 2024-10-16 13:46:10.006 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:46:10.006 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:46:10.025 INFO 4486 --- [Check obproxy1,a9ecba64b7fcefa5,02e94c0751504de9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:46:10.026 INFO 4486 --- [Check obproxy1,a9ecba64b7fcefa5,02e94c0751504de9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:10.026 INFO 4486 --- [Check obproxy2,93d7946c8c08835d,2b9a47e5e7a0b344] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:46:10.026 INFO 4486 --- [Check obproxy2,93d7946c8c08835d,2b9a47e5e7a0b344] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:10.031 INFO 4486 --- [Collect obproxy2,7e15c2e21e1b1094,fc3dad17f2915e56] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:46:10.031 INFO 4486 --- [Collect obproxy2,7e15c2e21e1b1094,fc3dad17f2915e56] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:10.031 INFO 4486 --- [Collect obproxy1,fa9b7f7f102ada81,93e4a7c71bacd133] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:46:10.031 INFO 4486 --- [Collect obproxy1,fa9b7f7f102ada81,93e4a7c71bacd133] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:10.052 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:10.054 INFO 4486 --- [Check obproxy2,93d7946c8c08835d,2b9a47e5e7a0b344] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:46:10.054 INFO 4486 --- [Check obproxy2,93d7946c8c08835d,2b9a47e5e7a0b344] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:46:10.055 INFO 4486 --- [Collect obproxy2,7e15c2e21e1b1094,fc3dad17f2915e56] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:46:10.055 INFO 4486 --- [Check obproxy1,a9ecba64b7fcefa5,02e94c0751504de9] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:46:10.055 INFO 4486 --- [Check obproxy1,a9ecba64b7fcefa5,02e94c0751504de9] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:46:10.056 INFO 4486 --- [Collect obproxy1,fa9b7f7f102ada81,93e4a7c71bacd133] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:46:10.060 INFO 4486 --- [Collect obproxy2,7e15c2e21e1b1094,fc3dad17f2915e56] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:46:10.061 INFO 4486 --- [Collect obproxy1,fa9b7f7f102ada81,93e4a7c71bacd133] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:46:10.065 INFO 4486 --- [Collect obproxy2,7e15c2e21e1b1094,fc3dad17f2915e56] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:46:10.065 INFO 4486 --- [Collect obproxy1,fa9b7f7f102ada81,93e4a7c71bacd133] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:46:10.751 INFO 4486 --- [ocp-schedules-5,20061a0643656b55,c1c6267d970b69cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:46:10.982 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:46:10.994 INFO 4486 --- [prometheus-minute-schedule-1,495bae995980ca7c,0f209290ab9f79c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:11.001 INFO 4486 --- [prometheus-minute-schedule-1,495bae995980ca7c,0f209290ab9f79c5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:11.007 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:46:11.270 INFO 4486 --- [ocp-async-7,e99058667dc83845,36c67a9c06fbe2be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:11.404 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f7f9094bbe2a101a,b05533bc80901e2a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f7f9094bbe2a101a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:11.405 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f7f9094bbe2a101a,b05533bc80901e2a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f7f9094bbe2a101a, duration=6 ms] 2024-10-16 13:46:11.684 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e8369caf392fe287,23d2e12603a2e13c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e8369caf392fe287, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:11.685 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e8369caf392fe287,23d2e12603a2e13c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e8369caf392fe287, duration=2 ms] 2024-10-16 13:46:13.006 INFO 4486 --- [ocp-async-9,f099aaf871dac962,060b499a24e664a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:46:13.054 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:13.270 INFO 4486 --- [ocp-async-6,758d75f16a7732b3,6150a690c99ee59d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:13.528 INFO 4486 --- [ocp-schedules-17,feb0ab94633f21eb,8c364a6201493502] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:46:13.866 ERROR 4486 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2457395) Duplicate entry '586774769288417280-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.865841] [YB420AC531B8-00060B7DA1ABDEE5-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2457395) Duplicate entry '586774769288417280-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.865841] [YB420AC531B8-00060B7DA1ABDEE5-0-0] 2024-10-16 13:46:13.873 ERROR 4486 --- [metric-archive-second-29,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2323956) Duplicate entry '586679390278586368-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.872236] [YB420AC531B8-00060B7DBC3872D2-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '586679390278586368-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.872236] [YB420AC531B8-00060B7DBC3872D2-0-0] 2024-10-16 13:46:13.873 ERROR 4486 --- [metric-archive-second-17,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620709) Duplicate entry '586679390194700288-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.872787] [YB420AC531B8-00060B7DB46FEF80-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620709) Duplicate entry '586679390194700288-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.872787] [YB420AC531B8-00060B7DB46FEF80-0-0] 2024-10-16 13:46:13.875 ERROR 4486 --- [metric-archive-second-21,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '587073990944165888-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.874273] [YB420AC531B8-00060B7DB03B2661-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '587073990944165888-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.874273] [YB420AC531B8-00060B7DB03B2661-0-0] 2024-10-16 13:46:13.875 ERROR 4486 --- [metric-archive-second-27,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=70525) Duplicate entry '587073990843502592-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.874557] [YB420AC531B8-00060B7DB7688596-0-0]; nested exception is java.sql.BatchUpdateException: (conn=70525) Duplicate entry '587073990843502592-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.874557] [YB420AC531B8-00060B7DB7688596-0-0] 2024-10-16 13:46:13.875 ERROR 4486 --- [metric-archive-second-11,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1525855) Duplicate entry '586679389997568000-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.874962] [YB420AC531B8-00060B7DB4196444-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1525855) Duplicate entry '586679389997568000-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.874962] [YB420AC531B8-00060B7DB4196444-0-0] 2024-10-16 13:46:13.875 ERROR 4486 --- [metric-archive-second-19,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '586774769053536256-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.875119] [YB420AC531B8-00060B7D84EAB11A-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '586774769053536256-1729057500' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:46:13.875119] [YB420AC531B8-00060B7D84EAB11A-0-0] 2024-10-16 13:46:13.994 INFO 4486 --- [prometheus-minute-schedule-1,8133a056b6697af6,314f8931195b7a01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:14.003 INFO 4486 --- [prometheus-minute-schedule-1,8133a056b6697af6,314f8931195b7a01] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:14.099 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057500, nodeCount=302, affectRows=302 2024-10-16 13:46:14.101 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057500, nodeCount=302, validCount=302 2024-10-16 13:46:14.227 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057500, nodeCount=463, affectRows=463 2024-10-16 13:46:14.229 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057500, nodeCount=463, validCount=463 2024-10-16 13:46:14.236 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057500, nodeCount=462, affectRows=462 2024-10-16 13:46:14.238 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057500, nodeCount=462, validCount=462 2024-10-16 13:46:14.249 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057500, nodeCount=518, affectRows=518 2024-10-16 13:46:14.251 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057500, nodeCount=518, validCount=518 2024-10-16 13:46:14.268 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057500, nodeCount=553, affectRows=553 2024-10-16 13:46:14.270 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057500, nodeCount=552, validCount=552 2024-10-16 13:46:14.278 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057500, nodeCount=538, affectRows=538 2024-10-16 13:46:14.280 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057500, nodeCount=537, validCount=537 2024-10-16 13:46:14.281 INFO 4486 --- [ocp-schedules-4,c0a3c32831c38e74,b2e8dd2adf6b2d0d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:46:14.307 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057500, nodeCount=614, affectRows=614 2024-10-16 13:46:14.308 WARN 4486 --- [scanner-13282401-1,c0a3c32831c38e74,d61135166513b86b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:46:14.309 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057500, nodeCount=613, validCount=613 2024-10-16 13:46:14.355 INFO 4486 --- [alarm-detect-1,c0a3c32831c38e74,71f2df529c9f8ce0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057545 ~ 1729057564 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057560, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057560, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057560=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:46:14.821 INFO 4486 --- [metric-archive-async-31,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057500, nodeCount=232, affectRows=232 2024-10-16 13:46:14.821 INFO 4486 --- [metric-archive-async-27,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057500, nodeCount=245, affectRows=245 2024-10-16 13:46:15.160 INFO 4486 --- [metric-archive-async-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057500, nodeCount=232, validCount=232 2024-10-16 13:46:15.179 INFO 4486 --- [metric-archive-async-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057500, nodeCount=245, validCount=245 2024-10-16 13:46:15.270 INFO 4486 --- [ocp-async-4,38f84892291d09d6,f0a511314e160d46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:16.055 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:16.994 INFO 4486 --- [prometheus-minute-schedule-1,763d761bf3aca739,07e4c9b7a8ec47d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:17.000 INFO 4486 --- [prometheus-minute-schedule-1,763d761bf3aca739,07e4c9b7a8ec47d4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:17.070 INFO 4486 --- [ocp-schedules-13,d87872fb50545328,95b093ed23a6ac8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:46:17.170 INFO 4486 --- [ocp-schedules-7,612c8224692a1825,1fe63d7659cd55e2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:46:17.184 INFO 4486 --- [ocp-schedules-19,2113b9e242a16763,2008f35f1026373e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:46:17.271 INFO 4486 --- [ocp-async-9,5af9b0a41d38c4c0,308d05515e1a4697] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:17.641 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3a6afcffdea0efd1,d23685a38b887f82] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3a6afcffdea0efd1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:17.641 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3a6afcffdea0efd1,d23685a38b887f82] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3a6afcffdea0efd1, duration=1 ms] 2024-10-16 13:46:18.287 INFO 4486 --- [ocp-schedules-16,66a6f06ea9273007,be2f5af3b71b70eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:46:18.287 INFO 4486 --- [ocp-schedules-16,438123d86ccfcc5c,2c5582230075e789] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:46:18.289 INFO 4486 --- [ocp-schedules-16,438123d86ccfcc5c,2c5582230075e789] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:18.315 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.316 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:18.328 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.329 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:18.348 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.349 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.349 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:46:18.350 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.351 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:18.376 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.377 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.377 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:18.378 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:46:18.379 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.382 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:46:18.383 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:18.392 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.392 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:18.402 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.412 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.413 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.413 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:18.415 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:46:18.421 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:46:18.425 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:18.426 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:18.450 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:46:18.458 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.460 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.461 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:46:18.506 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:46:18.506 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:46:18.508 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.509 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.510 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:18.514 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.518 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:46:18.521 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:18.529 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:46:18.539 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.540 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.541 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:46:18.548 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.548 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:18.552 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:46:18.552 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:46:18.579 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.579 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:18.608 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.608 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:18.635 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.636 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:18.660 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.660 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:18.686 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:18.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.712 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.713 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:18.717 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:46:18.722 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:46:18.727 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:46:18.728 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:46:18.753 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:46:18.792 INFO 4486 --- [ocp-schedules-5,a3c122d9220c75a0,6eb8c46eca27100f] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:46:18.794 INFO 4486 --- [ocp-schedules-5,a3c122d9220c75a0,6eb8c46eca27100f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:18.880 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:46:18.904 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:18.906 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:18.907 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:46:18.933 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:46:18.933 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:46:19.040 INFO 4486 --- [ocp-schedules-25,1412d00091770760,879619c1a8b0ce4e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:46:19.056 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:19.271 INFO 4486 --- [ocp-async-1,9ea6d2e39ffe288c,67bbfb1459ba0c1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:19.787 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ce17bcbb9047d06d,7d3347cb50863e40] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ce17bcbb9047d06d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:19.787 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ce17bcbb9047d06d,7d3347cb50863e40] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ce17bcbb9047d06d, duration=1 ms] 2024-10-16 13:46:19.994 INFO 4486 --- [prometheus-minute-schedule-1,edcea9f25eac194b,70f1faa42b5dba40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:20.000 INFO 4486 --- [prometheus-minute-schedule-1,edcea9f25eac194b,70f1faa42b5dba40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:20.756 INFO 4486 --- [ocp-schedules-7,91c25f17e68a7f36,a9ec660d1aa1f5a6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:46:20.812 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:46:20.996 INFO 4486 --- [prometheus-cache-monitor1,896d1b51f3d61951,3d1196f9b6b361f3] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13803,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8356} 2024-10-16 13:46:20.996 INFO 4486 --- [prometheus-cache-monitor1,896d1b51f3d61951,3d1196f9b6b361f3] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:46:21.271 INFO 4486 --- [ocp-async-6,3e0840b2d2ce1be2,59187f8306c48b15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:21.412 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a97c7bdd35b4abee,f7f530b5dcd9ca3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a97c7bdd35b4abee, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:21.412 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a97c7bdd35b4abee,f7f530b5dcd9ca3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a97c7bdd35b4abee, duration=1 ms] 2024-10-16 13:46:21.692 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,86f0a7ac86bd8aaf,446039b909ca5a7c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=86f0a7ac86bd8aaf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:21.693 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,86f0a7ac86bd8aaf,446039b909ca5a7c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=86f0a7ac86bd8aaf, duration=2 ms] 2024-10-16 13:46:22.057 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:22.994 INFO 4486 --- [prometheus-minute-schedule-1,cc7dbac9738dd5fb,fc2001dfccc09998] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:23.002 INFO 4486 --- [prometheus-minute-schedule-1,cc7dbac9738dd5fb,fc2001dfccc09998] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:23.006 INFO 4486 --- [ocp-async-2,9523dc08f70e9e1f,ccddf67f02d1657a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:46:23.271 INFO 4486 --- [ocp-async-3,98dea83c73b8d4a0,31a8db9821d70faa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:23.389 INFO 4486 --- [ocp-schedules-1,ff7a8e38cf86a4b6,5a0b30afbcb50858] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:46:24.410 INFO 4486 --- [ocp-schedules-28,db233bc1b3f2699e,13f61c4f137fe5be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:46:24.443 WARN 4486 --- [scanner-13282618-1,db233bc1b3f2699e,ea262792b6e474c7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:46:24.697 INFO 4486 --- [ocp-schedules-21,7f56374cb08eb8e8,bd56fd20e871f8ea] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:46:25.058 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:25.271 INFO 4486 --- [ocp-async-10,636cbfaf9db2bb72,1c226920b94edec5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:25.517 INFO 4486 --- [ocp-schedules-14,25a1d2daaffece09,3549ac7cba96bf8c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:46:25.517 INFO 4486 --- [ocp-schedules-14,25a1d2daaffece09,3549ac7cba96bf8c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:46:25.551 INFO 4486 --- [Collect all tenants major compactions3,25a1d2daaffece09,45de5ea81bfb9811] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:46:25.551 INFO 4486 --- [Collect all tenants major compactions5,25a1d2daaffece09,34e214220f1700ce] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:46:25.552 INFO 4486 --- [Collect all tenants major compactions2,25a1d2daaffece09,223f123f4227e58c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:46:25.552 INFO 4486 --- [Collect all tenants major compactions1,25a1d2daaffece09,ac878ae915339f83] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:46:25.552 INFO 4486 --- [Collect all tenants major compactions4,25a1d2daaffece09,c83da13bca2b92bb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:46:25.564 INFO 4486 --- [Collect all tenants major compactions1,25a1d2daaffece09,ac878ae915339f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:25.565 INFO 4486 --- [Collect all tenants major compactions2,25a1d2daaffece09,223f123f4227e58c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:25.566 INFO 4486 --- [Collect all tenants major compactions1,25a1d2daaffece09,ac878ae915339f83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:46:25.566 INFO 4486 --- [Collect all tenants major compactions2,25a1d2daaffece09,223f123f4227e58c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:46:25.570 INFO 4486 --- [Collect all tenants major compactions1,25a1d2daaffece09,ac878ae915339f83] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:46:25.570 INFO 4486 --- [Collect all tenants major compactions2,25a1d2daaffece09,223f123f4227e58c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:46:25.571 INFO 4486 --- [Collect all tenants major compactions2,25a1d2daaffece09,223f123f4227e58c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:46:25.571 INFO 4486 --- [Collect all tenants major compactions1,25a1d2daaffece09,ac878ae915339f83] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:46:25.573 INFO 4486 --- [Collect all tenants major compactions3,25a1d2daaffece09,45de5ea81bfb9811] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:25.575 INFO 4486 --- [Collect all tenants major compactions3,25a1d2daaffece09,45de5ea81bfb9811] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:46:25.578 INFO 4486 --- [Collect all tenants major compactions3,25a1d2daaffece09,45de5ea81bfb9811] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:46:25.578 INFO 4486 --- [Collect all tenants major compactions3,25a1d2daaffece09,45de5ea81bfb9811] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:46:25.580 INFO 4486 --- [Collect all tenants major compactions5,25a1d2daaffece09,34e214220f1700ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:25.581 INFO 4486 --- [Collect all tenants major compactions4,25a1d2daaffece09,c83da13bca2b92bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:25.582 INFO 4486 --- [Collect all tenants major compactions5,25a1d2daaffece09,34e214220f1700ce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:46:25.584 INFO 4486 --- [Collect all tenants major compactions4,25a1d2daaffece09,c83da13bca2b92bb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:46:25.586 INFO 4486 --- [Collect all tenants major compactions5,25a1d2daaffece09,34e214220f1700ce] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:46:25.586 INFO 4486 --- [Collect all tenants major compactions5,25a1d2daaffece09,34e214220f1700ce] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:46:25.587 INFO 4486 --- [Collect all tenants major compactions4,25a1d2daaffece09,c83da13bca2b92bb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:46:25.587 INFO 4486 --- [Collect all tenants major compactions4,25a1d2daaffece09,c83da13bca2b92bb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:46:25.588 INFO 4486 --- [ocp-schedules-14,25a1d2daaffece09,3549ac7cba96bf8c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:46:25.994 INFO 4486 --- [prometheus-minute-schedule-1,f0a38db605fcfe6a,f5d81037f5285d7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:25.999 INFO 4486 --- [prometheus-minute-schedule-1,f0a38db605fcfe6a,f5d81037f5285d7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:26.153 INFO 4486 --- [ocp-schedules-1,651f5b29e1969901,ea33b1c637fb8b1e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:46:26.155 INFO 4486 --- [ocp-schedules-1,651f5b29e1969901,ea33b1c637fb8b1e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:26.191 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:46:26.192 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.196 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:46:26.204 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.204 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:46:26.208 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:46:26.212 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.214 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.214 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:46:26.216 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:46:26.222 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.222 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:46:26.230 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.230 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:46:26.232 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.232 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:46:26.238 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.238 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:26.242 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.243 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:46:26.246 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:46:26.246 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.246 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.247 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:26.251 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:46:26.254 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.254 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.255 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:46:26.255 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:26.261 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.262 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:46:26.263 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.264 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:46:26.265 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.266 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:26.273 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.274 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:46:26.276 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.277 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:46:26.282 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.283 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.284 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.285 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.285 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.285 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.286 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:46:26.286 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.287 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.287 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.288 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.288 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:46:26.288 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.289 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.289 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.290 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:46:26.290 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.291 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.291 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.292 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.293 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.294 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.295 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.296 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.297 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.298 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.299 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.299 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.300 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.300 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.301 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.301 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:46:26.301 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:46:26.304 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:46:26.311 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.312 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:46:26.314 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.315 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:26.316 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.317 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.321 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.321 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:26.322 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.322 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.322 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:46:26.324 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.324 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.325 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:26.326 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.329 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.331 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:46:26.332 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.333 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.334 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.334 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:26.335 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:46:26.335 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:46:26.335 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.338 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.339 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.342 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.343 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.346 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.348 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.351 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.352 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.354 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:46:26.356 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:46:26.363 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.364 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.367 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.368 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.371 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.372 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.374 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.376 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.378 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.380 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.382 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.383 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.384 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:26.385 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.387 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:46:26.387 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.388 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:46:26.391 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.392 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:46:26.395 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.396 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:46:26.397 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.397 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:46:26.397 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.397 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:46:26.397 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.397 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:46:26.399 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:46:26.401 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:46:26.401 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:46:26.401 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:46:26.458 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:26.460 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:26.461 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:46:26.470 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.470 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:46:26.470 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.470 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:46:26.470 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:46:26.470 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:46:26.473 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:46:26.473 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:46:27.272 INFO 4486 --- [ocp-async-8,87b439a11de8abd3,4ccd5bfb2233af13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:27.344 INFO 4486 --- [ocp-schedules-26,670ba8847e47893c,2e0a725a585db0e2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:46:27.344+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:46:27.346 INFO 4486 --- [ocp-schedules-26,670ba8847e47893c,2e0a725a585db0e2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:46:27.347 INFO 4486 --- [ocp-schedules-26,670ba8847e47893c,2e0a725a585db0e2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:46:27.347 INFO 4486 --- [ocp-schedules-26,670ba8847e47893c,2e0a725a585db0e2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:46:27.649 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,708b407118d5e4a9,6aff63eec3f05e4a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=708b407118d5e4a9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:27.649 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,708b407118d5e4a9,6aff63eec3f05e4a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=708b407118d5e4a9, duration=3 ms] 2024-10-16 13:46:28.059 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:28.821 INFO 4486 --- [ocp-schedules-5,a3c122d9220c75a0,6eb8c46eca27100f] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:46:28.994 INFO 4486 --- [prometheus-minute-schedule-1,5b3a78c5e2943290,1d8fb002c15d9aa1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:29.002 INFO 4486 --- [prometheus-minute-schedule-1,5b3a78c5e2943290,1d8fb002c15d9aa1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:29.150 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-10-16 13:46:29.153 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=observer, size=5 2024-10-16 13:46:29.154 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=obproxy, size=3 2024-10-16 13:46:29.155 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=observer, size=5 2024-10-16 13:46:29.156 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=obproxy, size=3 2024-10-16 13:46:29.157 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=observer, size=5 2024-10-16 13:46:29.158 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=obproxy, size=3 2024-10-16 13:46:29.159 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=observer, size=5 2024-10-16 13:46:29.160 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=obproxy, size=3 2024-10-16 13:46:29.161 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=observer, size=5 2024-10-16 13:46:29.162 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=obproxy, size=3 2024-10-16 13:46:29.163 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=observer, size=5 2024-10-16 13:46:29.164 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=obproxy, size=3 2024-10-16 13:46:29.165 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=observer, size=5 2024-10-16 13:46:29.166 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=obproxy, size=3 2024-10-16 13:46:29.166 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-10-16 13:46:29.166 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=40024 2024-10-16 13:46:29.272 INFO 4486 --- [ocp-async-4,51c76ab3a7093020,0952a23854f89b97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703038633818001409 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703038633818001408 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703038633822195712 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703038633826390017 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703038633826390016 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703038633826390018 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703038633830584321 2024-10-16 13:46:29.291 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703038633830584320 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703038633834778624 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703038633838972929 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703038633838972928 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703038633838972930 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703038633843167233 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703038633843167232 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703038633847361536 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703038633851555841 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703038633851555840 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703038633855750145 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703038633855750144 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703038633855750146 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703038633859944448 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703038633864138752 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703038633872527360 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703038633880915968 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703038633889304576 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703038633897693184 2024-10-16 13:46:29.292 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703038633901887488 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703038633734115328 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703038633742503936 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703038633750892544 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703038633755086849 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703038633755086848 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703038633759281152 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703038633763475456 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703038633767669761 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703038633767669760 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703038633771864065 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703038633771864064 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703038633776058369 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703038633776058368 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703038633776058370 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703038633780252672 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703038633784446977 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703038633784446976 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703038633788641281 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703038633788641280 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703038633792835584 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703038633797029889 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703038633797029888 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703038633797029890 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703038633801224193 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703038633801224192 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703038633805418496 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703038633809612801 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703038633809612800 2024-10-16 13:46:29.637 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703038633813807104 2024-10-16 13:46:29.793 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,13b775dc20251bdb,b34b50c929f47fea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=13b775dc20251bdb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:29.793 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,13b775dc20251bdb,b34b50c929f47fea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=13b775dc20251bdb, duration=1 ms] 2024-10-16 13:46:30.103 INFO 4486 --- [prometheus-second-schedule-1,dc4200b73cbb75f7,cc86b0882a54174c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:46:30.762 INFO 4486 --- [ocp-schedules-21,d043139e212a275b,49fbf5d648d9e8cf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:46:31.060 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:31.272 INFO 4486 --- [ocp-async-3,2c2ea46385bfd42e,97a3e691dcca33cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:31.389 INFO 4486 --- [ocp-schedules-19,93ef314ea87b90fc,1776559c153e8c08] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=40024 2024-10-16 13:46:31.417 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,91554d954ac4f797,5932d1a60d0e5533] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=91554d954ac4f797, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:31.417 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,91554d954ac4f797,5932d1a60d0e5533] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=91554d954ac4f797, duration=1 ms] 2024-10-16 13:46:31.698 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,24d7c679faf68a74,c510429cf7c41760] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=24d7c679faf68a74, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:31.698 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,24d7c679faf68a74,c510429cf7c41760] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=24d7c679faf68a74, duration=1 ms] 2024-10-16 13:46:31.994 INFO 4486 --- [prometheus-minute-schedule-1,236adce1f36715b7,87e97035564b7dee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:32.000 INFO 4486 --- [prometheus-minute-schedule-1,236adce1f36715b7,87e97035564b7dee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:33.006 INFO 4486 --- [ocp-async-5,5c7db05b968924ed,bcf90286f0a29739] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:46:33.189 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a4756b9c579375a7,688a640ce16029ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=a4756b9c579375a7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:46:33.190 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a4756b9c579375a7,688a640ce16029ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=a4756b9c579375a7, duration=2 ms] 2024-10-16 13:46:33.272 INFO 4486 --- [ocp-async-2,fee669e85106aeef,be66065f87f940c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:34.060 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:34.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9e6612fffc3e55e6,5c313d63b7ea656b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=9e6612fffc3e55e6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:46:34.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9e6612fffc3e55e6,5c313d63b7ea656b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=9e6612fffc3e55e6, duration=2 ms] 2024-10-16 13:46:34.610 INFO 4486 --- [ocp-schedules-10,01097dd7f1f8e215,4014964e496dff98] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:46:34.640 WARN 4486 --- [scanner-13282957-1,01097dd7f1f8e215,780d463879d1e003] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:46:34.712 INFO 4486 --- [alarm-detect-8,01097dd7f1f8e215,1a7dcc97a5c566ad] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057560 ~ 1729057584 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057575, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057575, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057575=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:46:34.994 INFO 4486 --- [prometheus-minute-schedule-1,5267dd8d415abdab,87514160739c12ca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:35.001 INFO 4486 --- [prometheus-minute-schedule-1,5267dd8d415abdab,87514160739c12ca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:35.272 INFO 4486 --- [ocp-async-7,5d4bc3dd263b31ca,8ad014d3930485e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:37.062 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:37.063 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:46:37.064 INFO 4486 --- [ocp-schedules-14,733cc05eea98f912,c3b00571df0d0788] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:46:37.066 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:46:37.067 INFO 4486 --- [ocp-schedules-10,7e919594551bf78c,b1ef1ea65642b6d2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:46:37.068 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:46:37.069 INFO 4486 --- [ocp-schedules-14,733cc05eea98f912,c3b00571df0d0788] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:46:37.070 INFO 4486 --- [ocp-schedules-10,7e919594551bf78c,b1ef1ea65642b6d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:37.082 INFO 4486 --- [ocp-schedules-23,be8114c41121a6ce,cd9ac4ea91b7f18c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:46:37.083 INFO 4486 --- [ocp-schedules-12,5a95dd9df2d70777,ac03087c39a36be3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 13:46:37.083 INFO 4486 --- [ocp-schedules-23,be8114c41121a6ce,cd9ac4ea91b7f18c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:46:37.084 INFO 4486 --- [ocp-schedules-12,5a95dd9df2d70777,ac03087c39a36be3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:37.090 INFO 4486 --- [ocp-schedules-24,f95f4cea132d397d,89c6134c59ffc9d0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:46:37.090 INFO 4486 --- [ocp-schedules-4,59a1206514283854,b126317cbd58750f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:46:37.090 INFO 4486 --- [host-check-agent-status1,f95f4cea132d397d,9509baa3feebb790] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:46:37.090 INFO 4486 --- [host-check-agent-status2,f95f4cea132d397d,7cdc21d36e9f5373] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:46:37.090 INFO 4486 --- [host-check-agent-status3,f95f4cea132d397d,76adba367802c155] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:46:37.091 INFO 4486 --- [host-check-agent-status4,f95f4cea132d397d,da9b293e6a0d0267] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:46:37.091 INFO 4486 --- [ocp-schedules-19,6a6eadabf07b15d0,dfff442db1326ff8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:46:37.091 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:46:37.091 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:46:37.092 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:46:37.093 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:46:37.093 INFO 4486 --- [host-check-clockdiff4,59a1206514283854,75da745867f3c760] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:46:37.093 INFO 4486 --- [host-check-clockdiff2,59a1206514283854,98da2af5bf353f52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:46:37.093 INFO 4486 --- [host-check-clockdiff1,59a1206514283854,758b01f5cff14322] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:46:37.094 INFO 4486 --- [host-check-clockdiff3,59a1206514283854,fcd3194f86382388] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:46:37.103 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:37.104 INFO 4486 --- [host-check-agent-status2,f95f4cea132d397d,7cdc21d36e9f5373] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:46:37.107 INFO 4486 --- [host-check-agent-status3,f95f4cea132d397d,76adba367802c155] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:46:37.108 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:37.104 INFO 4486 --- [host-check-agent-status1,f95f4cea132d397d,9509baa3feebb790] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:37.109 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:46:37.109 INFO 4486 --- [host-check-agent-status3,f95f4cea132d397d,76adba367802c155] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:46:37.110 INFO 4486 --- [host-check-agent-status2,f95f4cea132d397d,7cdc21d36e9f5373] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:46:37.110 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:46:37.110 INFO 4486 --- [host-check-agent-status1,f95f4cea132d397d,9509baa3feebb790] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:37.110 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:46:37.111 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:46:37.112 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:46:37.112 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:46:37.114 INFO 4486 --- [host-check-agent-status4,f95f4cea132d397d,da9b293e6a0d0267] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:46:37.116 INFO 4486 --- [host-check-agent-status4,f95f4cea132d397d,da9b293e6a0d0267] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:46:37.123 INFO 4486 --- [host-check-clockdiff3,59a1206514283854,fcd3194f86382388] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057597 3 3 , error= 2024-10-16 13:46:37.142 INFO 4486 --- [host-check-clockdiff1,59a1206514283854,758b01f5cff14322] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057597 0 0 , error= 2024-10-16 13:46:37.155 INFO 4486 --- [host-check-clockdiff2,59a1206514283854,98da2af5bf353f52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057597 4 4 , error= 2024-10-16 13:46:37.157 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.158 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:37.159 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:46:37.160 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.160 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:46:37.168 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:46:37.171 INFO 4486 --- [host-check-agent-status3,f95f4cea132d397d,76adba367802c155] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:46:37.172 INFO 4486 --- [host-check-agent-status2,f95f4cea132d397d,7cdc21d36e9f5373] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:46:37.172 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:46:37.171597537+08:00 2024-10-16 13:46:37.172 INFO 4486 --- [host-check-agent-status1,f95f4cea132d397d,9509baa3feebb790] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:46:37.173 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:46:37.172+08:00 2024-10-16 13:46:37.174 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.175 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.175 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:37.176 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:46:37.176 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:46:37.172+08:00 and status AVAILABLE 2024-10-16 13:46:37.177 INFO 4486 --- [host-check-agent-status3,f95f4cea132d397d,76adba367802c155] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:46:37.177 INFO 4486 --- [host-check-agent-status3,f95f4cea132d397d,76adba367802c155] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:46:37.177 INFO 4486 --- [host-check-agent-status3,f95f4cea132d397d,76adba367802c155] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:46:37.177 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.178 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:46:37.178 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:46:37.178 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:46:37.181 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:46:37.182 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-agent-status1,f95f4cea132d397d,9509baa3feebb790] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-agent-status1,f95f4cea132d397d,9509baa3feebb790] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-agent-status1,f95f4cea132d397d,9509baa3feebb790] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:46:37.182794641+08:00 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-agent-status2,f95f4cea132d397d,7cdc21d36e9f5373] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-agent-status2,f95f4cea132d397d,7cdc21d36e9f5373] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-agent-status2,f95f4cea132d397d,7cdc21d36e9f5373] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:46:37.182 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:46:37.182 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.183 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:46:37.182+08:00 2024-10-16 13:46:37.183 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:46:37.184 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:46:37.187361410+08:00 2024-10-16 13:46:37.184 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:46:37.184 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:46:37.184+08:00 2024-10-16 13:46:37.185 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:46:37.188501153+08:00 2024-10-16 13:46:37.186 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:46:37.182+08:00 and status AVAILABLE 2024-10-16 13:46:37.186 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:46:37.185+08:00 2024-10-16 13:46:37.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b9897baaa20ebb4b,463a9313464fa595] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=b9897baaa20ebb4b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 13:46:37.187 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b9897baaa20ebb4b,463a9313464fa595] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=b9897baaa20ebb4b, duration=3 ms] 2024-10-16 13:46:37.188 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:46:37.184+08:00 and status AVAILABLE 2024-10-16 13:46:37.189 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:46:37.185+08:00 and status AVAILABLE 2024-10-16 13:46:37.189 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:46:37.189 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.190 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:46:37.190 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:46:37.190 INFO 4486 --- [host-check-agent-status4,f95f4cea132d397d,da9b293e6a0d0267] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:46:37.191 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:46:37.193 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.193 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.193 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.193 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.193 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.193 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.194 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.194 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.194 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:37.194 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:46:37.195 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:37.195 INFO 4486 --- [host-check-agent-status4,f95f4cea132d397d,da9b293e6a0d0267] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:46:37.195 INFO 4486 --- [host-check-agent-status4,f95f4cea132d397d,da9b293e6a0d0267] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:46:37.196 INFO 4486 --- [host-check-agent-status4,f95f4cea132d397d,da9b293e6a0d0267] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:46:37.197 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:46:37.197 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:46:37.198 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.199 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:46:37.199 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:46:37.200 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:46:37.201 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:46:37.201 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:46:37.201 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 13:46:37.205 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.206 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:37.208 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.212 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 13:46:37.212 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:46:37.215 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.216 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.217 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:37.218 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 13:46:37.220 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:37.221 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.222 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:37.223 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:37.224 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 13:46:37.227 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:46:37.229 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.230 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:46:37.232 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.233 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:46:37.241 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.242 INFO 4486 --- [ocp-schedules-30,03cea39e42be9fba,05c95807f491c14f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:46:37.255 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:46:37.255 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:46:37.260 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:46:37.267 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:46:37.270 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:46:37.270 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:46:37.273 INFO 4486 --- [ocp-async-3,65fadaf89a03bd0e,5e434ddc61d8257f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:37.279 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:46:37.285 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:37.286 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:37.287 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:37.291 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:46:37.306 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:46:37.314 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:46:37.317 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:46:37.325 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:46:37.327 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:46:37.328 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:46:37.337 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:46:37.341 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:37.346 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:37.358 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:37.362 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:46:37.369 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:46:37.374 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:46:37.379 INFO 4486 --- [host-check-host-status3,6a6eadabf07b15d0,d894d739bc853089] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:46:37.381 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:46:37.391 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:46:37.396 INFO 4486 --- [host-check-host-status4,6a6eadabf07b15d0,5ba82b515d78b11f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:46:37.396 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:46:37.405 INFO 4486 --- [host-check-host-status2,6a6eadabf07b15d0,9220060e26ae7bcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:46:37.443 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:37.445 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.446 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:46:37.447 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.448 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.448 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:46:37.449 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:46:37.458 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.458 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.458 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.458 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.458 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.458 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.458 INFO 4486 --- [host-check-host-status1,6a6eadabf07b15d0,604eef82ededc75c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:46:37.490 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:46:37.526 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:46:37.573 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:37.588 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:37.589 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.590 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:46:37.593 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:46:37.594 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.595 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:46:37.596 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.596 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.597 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:46:37.608 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.608 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.608 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.608 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.608 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.609 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.621 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:46:37.666 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,90142ee2928c6498,5955d0e5ff02d878] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=90142ee2928c6498, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:37.667 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,90142ee2928c6498,5955d0e5ff02d878] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=90142ee2928c6498, duration=1 ms] 2024-10-16 13:46:37.811 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:46:37.856 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:37.858 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.859 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:46:37.860 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.861 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.861 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:46:37.870 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.870 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.870 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.870 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.870 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:37.870 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:37.889 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:46:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:46:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:46:37.994 INFO 4486 --- [prometheus-minute-schedule-1,2aa5333980a75e09,1e9d1b0371d23a0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:37.996 INFO 4486 --- [ocp-schedules-22,4b76f49a10faefec,d08435a042e94646] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 13:46:37.996 INFO 4486 --- [ocp-schedules-22,4b76f49a10faefec,d08435a042e94646] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 13:46:38.000 INFO 4486 --- [prometheus-minute-schedule-1,2aa5333980a75e09,1e9d1b0371d23a0e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:38.033 INFO 4486 --- [pool-10-thread-6,1bf80c63334bebc3,7b2093d5faa4df1d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 13:46:38.033 INFO 4486 --- [pool-10-thread-6,1bf80c63334bebc3,7b2093d5faa4df1d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 13:46:38.033 INFO 4486 --- [pool-10-thread-5,1bf80c63334bebc3,a61be2491f51a854] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 13:46:38.035 INFO 4486 --- [pool-10-thread-8,1bf80c63334bebc3,5909430c06fa756e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 13:46:38.037 INFO 4486 --- [pool-10-thread-5,1bf80c63334bebc3,a61be2491f51a854] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 13:46:38.037 INFO 4486 --- [pool-10-thread-5,1bf80c63334bebc3,a61be2491f51a854] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 13:46:38.038 INFO 4486 --- [pool-10-thread-7,1bf80c63334bebc3,407cf8c5ebc29326] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 13:46:38.038 INFO 4486 --- [pool-overview-container-2,05182b4e3bbd6198,96fd6603f94eafe6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.040 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 13:46:38.040 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 13:46:38.041 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 13:46:38.041 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 13:46:38.041 INFO 4486 --- [pool-overview-container-2,05182b4e3bbd6198,96fd6603f94eafe6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:46:38.041 INFO 4486 --- [ForkJoinPool-6-worker-0,1bf80c63334bebc3,cf537b3686e6c725] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 13:46:38.041 INFO 4486 --- [ForkJoinPool-6-worker-0,1bf80c63334bebc3,cf537b3686e6c725] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 13:46:38.062 INFO 4486 --- [host-check-clockdiff4,59a1206514283854,75da745867f3c760] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057598 3 3 , error= 2024-10-16 13:46:38.073 INFO 4486 --- [ForkJoinPool-6-worker-0,1bf80c63334bebc3,cf537b3686e6c725] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.073 INFO 4486 --- [pool-10-thread-6,1bf80c63334bebc3,7b2093d5faa4df1d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 13:46:38.074 INFO 4486 --- [ForkJoinPool-6-worker-0,1bf80c63334bebc3,cf537b3686e6c725] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:46:38.075 INFO 4486 --- [ForkJoinPool-6-worker-0,1bf80c63334bebc3,cf537b3686e6c725] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.076 INFO 4486 --- [ForkJoinPool-6-worker-0,1bf80c63334bebc3,cf537b3686e6c725] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:46:38.080 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.080 INFO 4486 --- [pool-10-thread-6,1bf80c63334bebc3,7b2093d5faa4df1d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 13:46:38.082 INFO 4486 --- [ForkJoinPool-6-worker-11,1bf80c63334bebc3,ab50a90405e3e68a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.082 INFO 4486 --- [pool-overview-container-2,05182b4e3bbd6198,96fd6603f94eafe6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.082 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:46:38.084 INFO 4486 --- [ForkJoinPool-6-worker-11,1bf80c63334bebc3,ab50a90405e3e68a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:46:38.084 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.086 INFO 4486 --- [pool-overview-container-2,05182b4e3bbd6198,96fd6603f94eafe6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:46:38.089 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.090 WARN 4486 --- [ForkJoinPool-6-worker-0,1bf80c63334bebc3,cf537b3686e6c725] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:46:38.091 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:46:38.091 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:46:38.093 INFO 4486 --- [ForkJoinPool-6-worker-11,1bf80c63334bebc3,ab50a90405e3e68a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 13:46:38.094 INFO 4486 --- [ForkJoinPool-6-worker-11,1bf80c63334bebc3,ab50a90405e3e68a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.095 INFO 4486 --- [ForkJoinPool-6-worker-11,1bf80c63334bebc3,ab50a90405e3e68a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:46:38.096 INFO 4486 --- [ForkJoinPool-6-worker-11,1bf80c63334bebc3,ab50a90405e3e68a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.096 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.097 INFO 4486 --- [ForkJoinPool-6-worker-4,1bf80c63334bebc3,65cf43b7dad18c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.097 INFO 4486 --- [ForkJoinPool-6-worker-11,1bf80c63334bebc3,ab50a90405e3e68a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:46:38.097 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:46:38.099 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.100 INFO 4486 --- [ForkJoinPool-6-worker-4,1bf80c63334bebc3,65cf43b7dad18c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:46:38.102 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.103 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:46:38.105 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:46:38.110 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:46:38.112 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.114 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:46:38.118 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.119 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:46:38.123 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:46:38.123 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.126 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:46:38.128 INFO 4486 --- [ForkJoinPool-6-worker-4,1bf80c63334bebc3,65cf43b7dad18c5e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:46:38.129 INFO 4486 --- [ForkJoinPool-6-worker-4,1bf80c63334bebc3,65cf43b7dad18c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.131 INFO 4486 --- [ForkJoinPool-6-worker-4,1bf80c63334bebc3,65cf43b7dad18c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:46:38.134 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:38.135 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.137 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:46:38.137 INFO 4486 --- [ForkJoinPool-6-worker-4,1bf80c63334bebc3,65cf43b7dad18c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.139 INFO 4486 --- [ForkJoinPool-6-worker-4,1bf80c63334bebc3,65cf43b7dad18c5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:46:38.139 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.143 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.147 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:46:38.147 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:46:38.152 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.152 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.153 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:46:38.154 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:46:38.157 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:38.159 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.165 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:46:38.170 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.171 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:46:38.171 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:46:38.172 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.174 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:46:38.176 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.177 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:46:38.178 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.178 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:46:38.179 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:38.182 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.182 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 13:46:38.182 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:46:38.184 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:38.185 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.186 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:46:38.195 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:38.195 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:38.195 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:38.195 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:38.195 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:46:38.196 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.196 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.197 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:46:38.199 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.200 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:46:38.201 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:46:38.206 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.207 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:46:38.207 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.210 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.210 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:46:38, 2024-10-16 13:46:38] 2024-10-16 13:46:38.211 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 13:46:38.212 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:46:38.212 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.213 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.217 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:46:38.218 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 13:46:38.227 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:46:38.229 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.229 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.231 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:46:38.231 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:46:38.232 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.233 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.234 INFO 4486 --- [ForkJoinPool-6-worker-2,1bf80c63334bebc3,87ec9b628128b503] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:46:38.234 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:46:38, 2024-10-16 13:46:38] 2024-10-16 13:46:38.234 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:46:38.235 INFO 4486 --- [pool-10-thread-7,1bf80c63334bebc3,407cf8c5ebc29326] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 13:46:38.238 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 13:46:38.240 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.242 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:46:38, 2024-10-16 13:46:38] 2024-10-16 13:46:38.250 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:46:38.252 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.253 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:46:38, 2024-10-16 13:46:38] 2024-10-16 13:46:38.255 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 13:46:38.256 INFO 4486 --- [ForkJoinPool-6-worker-7,1bf80c63334bebc3,4ab1caae70c8c26b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 13:46:38.263 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:38.264 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:46:38.265 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.270 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 13:46:38.274 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 13:46:38.275 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.277 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:46:38, 2024-10-16 13:46:38] 2024-10-16 13:46:38.290 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:38.291 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.297 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:46:38.301 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:38.302 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.305 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:46:38.307 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:38.307 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:46:38.308 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:46:38, 2024-10-16 13:46:38] 2024-10-16 13:46:38.312 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 13:46:38.312 INFO 4486 --- [ForkJoinPool-6-worker-9,1bf80c63334bebc3,27bfc81167c03c05] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 13:46:38.315 INFO 4486 --- [pool-10-thread-8,1bf80c63334bebc3,5909430c06fa756e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 13:46:38.462 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:46:38.268426, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:46:38.462 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:46:38.268426, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:46:38.497 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:46:38.268426} } 2024-10-16 13:46:38.549 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:46:38.555 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:46:38.558 INFO 4486 --- [ocp-schedules-18,e0e06794460ed273,8c9ff4fb46d86806] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:46:39.045 INFO 4486 --- [ocp-schedules-13,deb66d5373ab00dd,5466ddf1abf5a406] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:46:39.273 INFO 4486 --- [ocp-async-8,a5e23db012bca7e2,04d8645da4fcfac9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:39.800 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1cc4711cd9b65a49,4f08465110614e0d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1cc4711cd9b65a49, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:39.800 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1cc4711cd9b65a49,4f08465110614e0d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1cc4711cd9b65a49, duration=1 ms] 2024-10-16 13:46:40.007 INFO 4486 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:46:40.007 INFO 4486 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:46:40.027 INFO 4486 --- [Check obproxy2,5b9cd87510656435,36ee87a9b5efa7bc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:46:40.027 INFO 4486 --- [Check obproxy2,5b9cd87510656435,36ee87a9b5efa7bc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:40.027 INFO 4486 --- [Check obproxy1,c27425398cfc4e0a,ac2878918fea16e4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:46:40.027 INFO 4486 --- [Check obproxy1,c27425398cfc4e0a,ac2878918fea16e4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:40.032 INFO 4486 --- [Collect obproxy2,88413672d1c68dbd,d7eb0226f5aa0e09] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:46:40.032 INFO 4486 --- [Collect obproxy2,88413672d1c68dbd,d7eb0226f5aa0e09] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:40.032 INFO 4486 --- [Collect obproxy1,b2c3efe7e762ca2e,40c25009ca1ad92f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:46:40.032 INFO 4486 --- [Collect obproxy1,b2c3efe7e762ca2e,40c25009ca1ad92f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:46:40.050 INFO 4486 --- [Check obproxy2,5b9cd87510656435,36ee87a9b5efa7bc] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:46:40.050 INFO 4486 --- [Check obproxy2,5b9cd87510656435,36ee87a9b5efa7bc] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:46:40.053 INFO 4486 --- [Collect obproxy2,88413672d1c68dbd,d7eb0226f5aa0e09] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:46:40.057 INFO 4486 --- [Check obproxy1,c27425398cfc4e0a,ac2878918fea16e4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:46:40.057 INFO 4486 --- [Check obproxy1,c27425398cfc4e0a,ac2878918fea16e4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:46:40.058 INFO 4486 --- [Collect obproxy2,88413672d1c68dbd,d7eb0226f5aa0e09] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:46:40.062 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:40.062 INFO 4486 --- [Collect obproxy2,88413672d1c68dbd,d7eb0226f5aa0e09] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:46:40.076 INFO 4486 --- [Collect obproxy1,b2c3efe7e762ca2e,40c25009ca1ad92f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:46:40.082 INFO 4486 --- [Collect obproxy1,b2c3efe7e762ca2e,40c25009ca1ad92f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:46:40.086 INFO 4486 --- [Collect obproxy1,b2c3efe7e762ca2e,40c25009ca1ad92f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:46:40.767 INFO 4486 --- [ocp-schedules-22,4083a872ad3da426,af6fdbd7c7287525] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:46:40.994 INFO 4486 --- [prometheus-minute-schedule-1,47cd4fc9aa54dbae,332573d6de19960f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:40.995 INFO 4486 --- [prometheus-hour-schedule-1,8a9d4bb8f336e944,5ddf5936842bc8e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:46:41.000 INFO 4486 --- [prometheus-minute-schedule-1,47cd4fc9aa54dbae,332573d6de19960f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:41.273 INFO 4486 --- [ocp-async-4,b175b92e23f95e79,7b15b292a19d0e93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:41.423 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,87d69947e4696129,8e03ded60bd4f271] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=87d69947e4696129, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:41.423 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,87d69947e4696129,8e03ded60bd4f271] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=87d69947e4696129, duration=1 ms] 2024-10-16 13:46:41.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,84644bd1d908231b,f6b273855ddd22e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=84644bd1d908231b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:41.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,84644bd1d908231b,f6b273855ddd22e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=84644bd1d908231b, duration=1 ms] 2024-10-16 13:46:42.128 INFO 4486 --- [ocp-schedules-30,544721be5e3fa411,b55b0dd920e73d0f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:46:43.006 INFO 4486 --- [ocp-async-7,414eb3e91ee52830,22aa48f5917ccaa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:46:43.063 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:43.273 INFO 4486 --- [ocp-async-10,c96862d54839a1b5,95a67f0298637a69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:43.994 INFO 4486 --- [prometheus-minute-schedule-1,2e1bec474f67dafc,293783e04792aefb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:44.002 INFO 4486 --- [prometheus-minute-schedule-1,2e1bec474f67dafc,293783e04792aefb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:44.788 INFO 4486 --- [ocp-schedules-20,533c5d045b53b1eb,34f1622923865eca] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:46:44.840 WARN 4486 --- [scanner-13283197-1,533c5d045b53b1eb,2e913997ba55f09c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:46:44.919 INFO 4486 --- [alarm-detect-5,533c5d045b53b1eb,8aaa723329f7f537] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057575 ~ 1729057594 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057590, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057590, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057590=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:46:45.274 INFO 4486 --- [ocp-async-2,38b8500521717359,ea20fb6f1147c5bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:45.516 INFO 4486 --- [ocp-schedules-27,19f0c4a986b3b9ca,b135f78b7d7b3840] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:46:45.520 INFO 4486 --- [ocp-schedules-27,19f0c4a986b3b9ca,b135f78b7d7b3840] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:46:45.520 INFO 4486 --- [ocp-schedules-27,19f0c4a986b3b9ca,b135f78b7d7b3840] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:46:45.537 INFO 4486 --- [ocp-schedules-22,fcb9352bef7401d7,eaa3dacb675005ea] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:46:45.537 INFO 4486 --- [ocp-schedules-22,fcb9352bef7401d7,eaa3dacb675005ea] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:46:46.064 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:46.994 INFO 4486 --- [prometheus-minute-schedule-1,9596b085e80b7477,2ea3c0ef42dc8f71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:47.001 INFO 4486 --- [prometheus-minute-schedule-1,9596b085e80b7477,2ea3c0ef42dc8f71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:47.072 INFO 4486 --- [ocp-schedules-10,7e919594551bf78c,b1ef1ea65642b6d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:46:47.174 INFO 4486 --- [ocp-schedules-11,33e6285c0364ba2f,f9c249af73cb145d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:46:47.182 INFO 4486 --- [ocp-schedules-23,be8114c41121a6ce,cd9ac4ea91b7f18c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:46:47.227 INFO 4486 --- [ocp-schedules-12,5a95dd9df2d70777,ac03087c39a36be3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 13:46:47.274 INFO 4486 --- [ocp-async-7,cceb674a78405426,84ecaf409fa2329d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:47.673 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,1fb9b3f1255aa3b8,9cbb58ccdcd33415] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1fb9b3f1255aa3b8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:47.673 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,1fb9b3f1255aa3b8,9cbb58ccdcd33415] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1fb9b3f1255aa3b8, duration=3 ms] 2024-10-16 13:46:48.933 INFO 4486 --- [ocp-schedules-16,438123d86ccfcc5c,2c5582230075e789] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:46:48.934 INFO 4486 --- [ocp-schedules-16,e94fdbba3d7be92a,59df651991b16cf1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:46:48.936 INFO 4486 --- [ocp-schedules-16,e94fdbba3d7be92a,59df651991b16cf1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:46:48.962 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:48.963 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:48.973 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:48.974 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:48.992 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:48.993 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:49.000 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.001 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.001 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:46:49.034 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.035 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.035 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:49.036 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:46:49.036 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.039 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:46:49.041 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:49.090 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:49.096 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.096 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:49.106 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.116 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.117 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.118 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:49.119 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:46:49.124 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:46:49.129 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:49.131 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:49.142 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.144 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.144 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:46:49.156 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:46:49.175 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:46:49.175 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:46:49.194 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.196 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.197 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:46:49.201 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.204 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:46:49.207 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:49.228 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:46:49.232 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.232 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:49.237 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.238 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.239 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:46:49.248 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:46:49.248 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:46:49.259 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.260 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:49.274 INFO 4486 --- [ocp-async-1,467f0f776c01e5dc,4d601d9f6f8247fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:49.281 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.281 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:49.306 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.306 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:46:49.331 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.331 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:46:49.365 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:46:49.393 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.395 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.396 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:46:49.400 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:46:49.406 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:46:49.411 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:46:49.413 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:46:49.442 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:46:49.584 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:46:49.613 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:46:49.615 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:46:49.616 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:46:49.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:46:49.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:46:49.805 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,41d8e561a32ec06e,dfd6c80a023f2744] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=41d8e561a32ec06e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:49.805 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,41d8e561a32ec06e,dfd6c80a023f2744] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=41d8e561a32ec06e, duration=1 ms] 2024-10-16 13:46:49.994 INFO 4486 --- [prometheus-minute-schedule-1,763eb90b45f2d5a7,172a563b6d0fbe3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:50.000 INFO 4486 --- [prometheus-minute-schedule-1,763eb90b45f2d5a7,172a563b6d0fbe3e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:50.461 INFO 4486 --- [ocp-schedules-5,43fdf6e6fd0e5327,228e4e80b1461867] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:46:50.461 INFO 4486 --- [ocp-schedules-5,43fdf6e6fd0e5327,228e4e80b1461867] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:46:50.462 INFO 4486 --- [ocp-schedules-5,43fdf6e6fd0e5327,228e4e80b1461867] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:46:50.462 INFO 4486 --- [ocp-schedules-5,43fdf6e6fd0e5327,228e4e80b1461867] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:46:50.462 INFO 4486 --- [ocp-schedules-5,43fdf6e6fd0e5327,228e4e80b1461867] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:46:50.462 INFO 4486 --- [ocp-schedules-5,43fdf6e6fd0e5327,228e4e80b1461867] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703038633901887488 2024-10-16 13:46:50.466 INFO 4486 --- [ocp-schedules-5,43fdf6e6fd0e5327,228e4e80b1461867] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=56, maxSyncedRecordId=703041150484287488 2024-10-16 13:46:50.664 INFO 4486 --- [ocp-schedules-3,db34b66c884ee7bd,6d141c904c958272] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:46:50.664 INFO 4486 --- [ocp-schedules-3,db34b66c884ee7bd,6d141c904c958272] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:46:50.664 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:46:50.664 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:46:50.664 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:46:50.664 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:46:50.664 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:46:50.669 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:46:50.669 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:46:50.669 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:46:50.670 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:46:50.670 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:46:50.671 INFO 4486 --- [ocp-schedules-3,db34b66c884ee7bd,6d141c904c958272] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:46:50.671 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:46:50.671 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:46:50.671 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:46:50.671 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:46:50.672 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:46:50.672 INFO 4486 --- [ocp-schedules-3,db34b66c884ee7bd,6d141c904c958272] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:46:50.694 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:46:50.694 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:46:50.698 INFO 4486 --- [ocp-schedules-3,db34b66c884ee7bd,6d141c904c958272] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:46:50.699 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:46:50.702 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:46:50.714 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:46:50.720 INFO 4486 --- [ocp-schedules-3,db34b66c884ee7bd,6d141c904c958272] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:46:50.774 INFO 4486 --- [ocp-schedules-24,f93846f076413584,53de38fc6e391165] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:46:51.274 INFO 4486 --- [ocp-async-6,29c719faf985b14c,2c8be653034cc693] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:51.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,5148a1937a9ba24f,5959e826a47aa656] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5148a1937a9ba24f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:51.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,5148a1937a9ba24f,5959e826a47aa656] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5148a1937a9ba24f, duration=1 ms] 2024-10-16 13:46:51.718 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,d1ac97725a43a560,ce95e9000a185cd8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d1ac97725a43a560, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:51.718 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,d1ac97725a43a560,ce95e9000a185cd8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d1ac97725a43a560, duration=1 ms] 2024-10-16 13:46:52.091 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:52.994 INFO 4486 --- [prometheus-minute-schedule-1,d67e62eead410281,9bf83cf3cf9c7607] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:52.999 INFO 4486 --- [prometheus-minute-schedule-1,d67e62eead410281,9bf83cf3cf9c7607] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:53.007 INFO 4486 --- [ocp-async-2,2ac89867e5a96299,4a249cb3bd899929] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:46:53.274 INFO 4486 --- [ocp-async-9,758df62787f39162,d363f6c87d28f5ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:53.380 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,28818f2d4c68a6ab,5325c5ba4af97046] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=28818f2d4c68a6ab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:46:53.380 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,28818f2d4c68a6ab,5325c5ba4af97046] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=28818f2d4c68a6ab, duration=1 ms] 2024-10-16 13:46:55.043 INFO 4486 --- [ocp-schedules-27,4c4b24565ccf2595,b739ad66e67dab8d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:46:55.082 WARN 4486 --- [scanner-13283551-1,4c4b24565ccf2595,9a57601268e4a28d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:46:55.092 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:55.120 INFO 4486 --- [alarm-detect-18,4c4b24565ccf2595,20c6123ecf64c5b3] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057590 ~ 1729057605 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057605, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057605, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057605=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:46:55.275 INFO 4486 --- [ocp-async-5,6b96fc5b2e1ebf69,53bde716c90a310d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:55.994 INFO 4486 --- [prometheus-minute-schedule-1,9e50ab8b55fbbf2e,98e176a2a393b55a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:55.999 INFO 4486 --- [prometheus-minute-schedule-1,9e50ab8b55fbbf2e,98e176a2a393b55a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:57.275 INFO 4486 --- [ocp-async-3,9fb18c612e300928,9df844744fba7aff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:57.679 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,b7d8a117c14ef5ea,a1d460b6113aeb56] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b7d8a117c14ef5ea, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:57.679 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,b7d8a117c14ef5ea,a1d460b6113aeb56] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b7d8a117c14ef5ea, duration=1 ms] 2024-10-16 13:46:58.093 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:46:58.994 INFO 4486 --- [prometheus-minute-schedule-1,900c1bf9a2e9201f,015447396b224caa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057620, interval=60 2024-10-16 13:46:59.000 INFO 4486 --- [prometheus-minute-schedule-1,900c1bf9a2e9201f,015447396b224caa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:46:59.050 INFO 4486 --- [ocp-schedules-12,87ecd4bfe483e380,05820a1bee68cd91] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:46:59.275 INFO 4486 --- [ocp-async-8,d877fdaebf6b45f2,64f701c5c0d7536b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:46:59.810 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,bb3f1a4dc82aa03f,96ec4f77acd2a0ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bb3f1a4dc82aa03f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:46:59.810 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,bb3f1a4dc82aa03f,96ec4f77acd2a0ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bb3f1a4dc82aa03f, duration=1 ms] 2024-10-16 13:47:00.778 INFO 4486 --- [ocp-schedules-14,d018312e5d849a6f,4f80da68858351d3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:47:01.094 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:01.275 INFO 4486 --- [ocp-async-9,5fd6d0b4a699d98e,7f25ea22350ffea7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:01.435 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,22a5c9b5ef46170f,d3577c14af1566e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=22a5c9b5ef46170f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:01.436 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,22a5c9b5ef46170f,d3577c14af1566e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=22a5c9b5ef46170f, duration=3 ms] 2024-10-16 13:47:01.722 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,d467210ef9e752d5,64563bb50c18bd59] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d467210ef9e752d5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:01.722 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,d467210ef9e752d5,64563bb50c18bd59] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d467210ef9e752d5, duration=1 ms] 2024-10-16 13:47:01.871 INFO 4486 --- [ocp-schedules-22,07bbe358e40109f8,b3ebad545156dd66] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:47:01.871+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:47:01.876 INFO 4486 --- [ocp-schedules-22,07bbe358e40109f8,b3ebad545156dd66] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:47:01.881 INFO 4486 --- [ocp-schedules-22,07bbe358e40109f8,b3ebad545156dd66] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:47:01.891 INFO 4486 --- [ocp-schedules-22,07bbe358e40109f8,b3ebad545156dd66] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster glob_3zone for slowSqlStat, sql count is 0. 2024-10-16 13:47:01.994 INFO 4486 --- [prometheus-minute-schedule-1,f43bfbf2d3e05616,40ea5b9c924895d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:02.000 INFO 4486 --- [prometheus-minute-schedule-1,f43bfbf2d3e05616,40ea5b9c924895d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:03.007 INFO 4486 --- [ocp-async-10,edb8f26cdc3fa325,533f0c013655291d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:47:03.275 INFO 4486 --- [ocp-async-2,3b6dd082683f28e6,b89c62e53ad0a864] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:04.095 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:04.994 INFO 4486 --- [prometheus-minute-schedule-1,8ee7a70ea2172790,d589f385b6ebd87f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:04.999 INFO 4486 --- [prometheus-minute-schedule-1,8ee7a70ea2172790,d589f385b6ebd87f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:05.183 INFO 4486 --- [ocp-schedules-7,2350727a937443dd,6da1a0388e38aea2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:47:05.215 WARN 4486 --- [scanner-13283768-1,2350727a937443dd,3040314b1d8d1141] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:47:05.276 INFO 4486 --- [ocp-async-8,16603ffa15958282,99029aaea73d8c9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:07.066 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:47:07.066 INFO 4486 --- [ocp-schedules-12,ac0fd78aff4fa1c3,608c0591766b0407] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:47:07.068 INFO 4486 --- [ocp-schedules-12,ac0fd78aff4fa1c3,608c0591766b0407] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:47:07.081 INFO 4486 --- [ocp-schedules-29,96e7960747b0289c,c114afe96b2176a1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:47:07.083 INFO 4486 --- [ocp-schedules-29,96e7960747b0289c,c114afe96b2176a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:47:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:47:07.089 INFO 4486 --- [ocp-schedules-2,c7ac9816536d405b,58d872cccea58597] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:47:07.089 INFO 4486 --- [ocp-schedules-26,806fed5f80d23883,e6dd9dd51b869372] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:47:07.089 INFO 4486 --- [ocp-schedules-18,a67e1cb417f03339,ffe3f781e9f15290] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:47:07.090 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:47:07.091 INFO 4486 --- [host-check-agent-status1,806fed5f80d23883,b959b6e482cc7c48] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:47:07.091 INFO 4486 --- [host-check-agent-status2,806fed5f80d23883,64e395f3d423e49d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:47:07.091 INFO 4486 --- [host-check-agent-status3,806fed5f80d23883,19fb7d967baf37cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:47:07.092 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:47:07.092 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:47:07.092 INFO 4486 --- [host-check-clockdiff1,c7ac9816536d405b,83b08f5050a18945] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:47:07.092 INFO 4486 --- [host-check-clockdiff2,c7ac9816536d405b,6ec674bb8d3ca945] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:47:07.093 INFO 4486 --- [host-check-agent-status4,806fed5f80d23883,430a2acac8423498] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:47:07.093 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:47:07.094 INFO 4486 --- [host-check-clockdiff4,c7ac9816536d405b,15061e58bcc9f693] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:47:07.095 INFO 4486 --- [host-check-clockdiff3,c7ac9816536d405b,b7235ca15e148379] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:47:07.096 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:07.100 INFO 4486 --- [host-check-agent-status1,806fed5f80d23883,b959b6e482cc7c48] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:07.100 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:07.101 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:07.101 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:07.100 INFO 4486 --- [host-check-agent-status3,806fed5f80d23883,19fb7d967baf37cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:07.101 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:47:07.102 INFO 4486 --- [host-check-agent-status4,806fed5f80d23883,430a2acac8423498] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:47:07.102 INFO 4486 --- [host-check-agent-status2,806fed5f80d23883,64e395f3d423e49d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:07.104 INFO 4486 --- [host-check-agent-status1,806fed5f80d23883,b959b6e482cc7c48] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:07.104 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:07.104 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:47:07.104 INFO 4486 --- [host-check-agent-status2,806fed5f80d23883,64e395f3d423e49d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:07.126 INFO 4486 --- [host-check-agent-status4,806fed5f80d23883,430a2acac8423498] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:47:07.127 INFO 4486 --- [host-check-agent-status3,806fed5f80d23883,19fb7d967baf37cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:07.127 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:07.127 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:07.133 INFO 4486 --- [host-check-clockdiff3,c7ac9816536d405b,b7235ca15e148379] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057627 4 4 , error= 2024-10-16 13:47:07.136 INFO 4486 --- [host-check-clockdiff1,c7ac9816536d405b,83b08f5050a18945] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057627 0 0 , error= 2024-10-16 13:47:07.136 INFO 4486 --- [host-check-clockdiff2,c7ac9816536d405b,6ec674bb8d3ca945] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057627 4 4 , error= 2024-10-16 13:47:07.142 INFO 4486 --- [host-check-clockdiff4,c7ac9816536d405b,15061e58bcc9f693] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057627 3 3 , error= 2024-10-16 13:47:07.155 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.156 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:07.157 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:47:07.170 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:47:07.170 INFO 4486 --- [host-check-agent-status1,806fed5f80d23883,b959b6e482cc7c48] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:47:07.170 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:47:07.170 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:47:07.171 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:47:07.172 INFO 4486 --- [host-check-agent-status3,806fed5f80d23883,19fb7d967baf37cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:47:07.176 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:47:07.172249788+08:00 2024-10-16 13:47:07.176 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:47:07.176947413+08:00 2024-10-16 13:47:07.176 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.176 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:47:07.176+08:00 2024-10-16 13:47:07.176 INFO 4486 --- [host-check-agent-status1,806fed5f80d23883,b959b6e482cc7c48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:47:07.177 INFO 4486 --- [host-check-agent-status1,806fed5f80d23883,b959b6e482cc7c48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:47:07.177 INFO 4486 --- [host-check-agent-status1,806fed5f80d23883,b959b6e482cc7c48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:47:07.177 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:47:07.176+08:00 2024-10-16 13:47:07.177 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:47:07.180656747+08:00 2024-10-16 13:47:07.177 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:47:07.180121090+08:00 2024-10-16 13:47:07.177 INFO 4486 --- [host-check-agent-status4,806fed5f80d23883,430a2acac8423498] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:47:07.177 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:07.177 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:47:07.177+08:00 2024-10-16 13:47:07.178 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:47:07.177+08:00 2024-10-16 13:47:07.179 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:47:07.176+08:00 and status AVAILABLE 2024-10-16 13:47:07.179 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:47:07.176+08:00 and status AVAILABLE 2024-10-16 13:47:07.179 INFO 4486 --- [host-check-agent-status2,806fed5f80d23883,64e395f3d423e49d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:47:07.180 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:47:07.177+08:00 and status AVAILABLE 2024-10-16 13:47:07.180 INFO 4486 --- [host-check-agent-status3,806fed5f80d23883,19fb7d967baf37cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:47:07.180 INFO 4486 --- [host-check-agent-status3,806fed5f80d23883,19fb7d967baf37cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:47:07.180 INFO 4486 --- [host-check-agent-status3,806fed5f80d23883,19fb7d967baf37cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:47:07.181 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:47:07.177+08:00 and status AVAILABLE 2024-10-16 13:47:07.181 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:47:07.181 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:47:07.182 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:47:07.182 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:47:07.183 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:47:07.183 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.184 INFO 4486 --- [host-check-agent-status2,806fed5f80d23883,64e395f3d423e49d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:47:07.184 INFO 4486 --- [host-check-agent-status2,806fed5f80d23883,64e395f3d423e49d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:47:07.184 INFO 4486 --- [host-check-agent-status2,806fed5f80d23883,64e395f3d423e49d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:47:07.185 INFO 4486 --- [host-check-agent-status4,806fed5f80d23883,430a2acac8423498] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:47:07.185 INFO 4486 --- [host-check-agent-status4,806fed5f80d23883,430a2acac8423498] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:47:07.185 INFO 4486 --- [host-check-agent-status4,806fed5f80d23883,430a2acac8423498] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:47:07.188 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:47:07.189 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:07.190 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:07.190 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:07.191 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:07.191 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:47:07.191 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:07.192 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:47:07.192 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:07.196 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.197 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:47:07.208 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.208 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:47:07.214 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:07.214 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:07.214 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:07.217 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:07.218 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.219 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:47:07.222 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.223 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:47:07.245 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:47:07.245 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:07.246 INFO 4486 --- [ocp-schedules-5,d62698d343c32297,04bb183519d0e713] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:47:07.250 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:47:07.254 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:47:07.256 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:47:07.262 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:47:07.266 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:47:07.273 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:07.276 INFO 4486 --- [ocp-async-4,574077ec245d9e86,af05848187bd4ab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:07.281 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:07.281 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:47:07.282 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:07.291 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:47:07.299 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:47:07.306 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:07.308 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:47:07.313 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:47:07.324 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:07.324 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:47:07.324 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:47:07.337 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:47:07.340 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:07.356 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:07.357 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:47:07.360 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:47:07.367 INFO 4486 --- [host-check-host-status3,a67e1cb417f03339,af9bdf78d7df178b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:47:07.373 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:47:07.374 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:47:07.384 INFO 4486 --- [host-check-host-status4,a67e1cb417f03339,6a6e546929eb86e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:47:07.389 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:07.394 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:47:07.406 INFO 4486 --- [host-check-host-status2,a67e1cb417f03339,aa6f64b3d5b7f53c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:47:07.451 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:47:07.462 INFO 4486 --- [host-check-host-status1,a67e1cb417f03339,c25c7a24eb40e130] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:47:07.644 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:47:07.644 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:47:07.653 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:47:07.653 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:47:07.653 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:47:07.653 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:47:07.660 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:47:07.660 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:47:07.660 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:47:07.660 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:47:07.685 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,29c9ee640a11092f,65bcb61431996d78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=29c9ee640a11092f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:07.685 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,29c9ee640a11092f,65bcb61431996d78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=29c9ee640a11092f, duration=2 ms] 2024-10-16 13:47:07.709 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:47:07.719 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:47:07.719 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:47:07.719 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:47:07.719 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:47:07.719 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:47:07.730 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:47:07.994 INFO 4486 --- [prometheus-minute-schedule-1,4792d9a25d03c1b6,9ee7407c9b52993e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:07.999 INFO 4486 --- [prometheus-minute-schedule-1,4792d9a25d03c1b6,9ee7407c9b52993e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:08.200 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:47:08.691 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.031340783S 2024-10-16 13:47:08.692 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:47:08.770 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:47:08.770 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:47:08.797 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:47:08.869 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:47:08.869 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:47:08.869 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:47:08.869 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:47:08.869 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:47:08.869 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:47:08.878 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:47:09.040 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] o.s.boot.SpringApplication : Started application in 1.387 seconds (JVM running for 78532.091) 2024-10-16 13:47:09.040 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.387555228S 2024-10-16 13:47:09.276 INFO 4486 --- [ocp-async-1,607eb077a9e086ea,10b33470bfca98e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:09.386 INFO 4486 --- [ocp-schedules-9,6d593ffded5e6b2c,a7688e09c37e74e1] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:47:09.816 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,1702d7cc5434586c,c67eb82b59f7acb1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1702d7cc5434586c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:09.816 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,1702d7cc5434586c,c67eb82b59f7acb1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1702d7cc5434586c, duration=2 ms] 2024-10-16 13:47:10.006 INFO 4486 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:47:10.007 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:47:10.037 INFO 4486 --- [Collect obproxy1,43a4ee8a273562cd,95d02e3e9cc50c2a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:47:10.038 INFO 4486 --- [Check obproxy1,12d67b0befa9e055,3bf46ff6b18a342a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:47:10.039 INFO 4486 --- [Check obproxy2,61072f05c5cbffff,e7af3b52d24b41c2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:47:10.039 INFO 4486 --- [Collect obproxy2,68bf0ef22fb20f88,73c42e5db58a06a2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:47:10.041 INFO 4486 --- [Collect obproxy1,43a4ee8a273562cd,95d02e3e9cc50c2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:10.044 INFO 4486 --- [Check obproxy2,61072f05c5cbffff,e7af3b52d24b41c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:10.047 INFO 4486 --- [Check obproxy1,12d67b0befa9e055,3bf46ff6b18a342a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:10.048 INFO 4486 --- [Collect obproxy2,68bf0ef22fb20f88,73c42e5db58a06a2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:10.066 INFO 4486 --- [Check obproxy2,61072f05c5cbffff,e7af3b52d24b41c2] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:47:10.066 INFO 4486 --- [Check obproxy2,61072f05c5cbffff,e7af3b52d24b41c2] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:47:10.068 INFO 4486 --- [Collect obproxy2,68bf0ef22fb20f88,73c42e5db58a06a2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:47:10.074 INFO 4486 --- [Collect obproxy2,68bf0ef22fb20f88,73c42e5db58a06a2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:47:10.078 INFO 4486 --- [Collect obproxy2,68bf0ef22fb20f88,73c42e5db58a06a2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:47:10.082 INFO 4486 --- [Collect obproxy1,43a4ee8a273562cd,95d02e3e9cc50c2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:47:10.083 INFO 4486 --- [Check obproxy1,12d67b0befa9e055,3bf46ff6b18a342a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:47:10.083 INFO 4486 --- [Check obproxy1,12d67b0befa9e055,3bf46ff6b18a342a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:47:10.086 INFO 4486 --- [Collect obproxy1,43a4ee8a273562cd,95d02e3e9cc50c2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:47:10.091 INFO 4486 --- [Collect obproxy1,43a4ee8a273562cd,95d02e3e9cc50c2a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:47:10.096 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:10.785 INFO 4486 --- [ocp-schedules-18,e2bb7c286a4fdfcf,a50105024df1f317] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:47:10.981 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:47:10.994 INFO 4486 --- [prometheus-minute-schedule-1,8e4ca298f93fc256,085e6c0f2ba41a3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:11.000 INFO 4486 --- [prometheus-minute-schedule-1,8e4ca298f93fc256,085e6c0f2ba41a3e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:47:11.276 INFO 4486 --- [ocp-async-2,42f6ae183bc50a0e,3672726e36bf5392] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:11.440 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,9589ac3b8a5f45a3,aa096a1ce918f4af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9589ac3b8a5f45a3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:11.440 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,9589ac3b8a5f45a3,aa096a1ce918f4af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9589ac3b8a5f45a3, duration=1 ms] 2024-10-16 13:47:11.750 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ca65764f20cae352,e19bfe14be8210dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ca65764f20cae352, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:11.751 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ca65764f20cae352,e19bfe14be8210dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ca65764f20cae352, duration=24 ms] 2024-10-16 13:47:13.007 INFO 4486 --- [ocp-async-7,57d05371dd56980a,7a6ae62aeae683f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:47:13.097 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:13.277 INFO 4486 --- [ocp-async-10,bc649547c3da4020,869e1bc393f1d496] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:13.528 INFO 4486 --- [ocp-schedules-7,33d3ffb50555ed79,0b1b2ef6443ad44a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:47:13.530 INFO 4486 --- [ocp-schedules-7,33d3ffb50555ed79,0b1b2ef6443ad44a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:47:13.531 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:47:13.531 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:47:13.531 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:47:13.939 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057560, nodeCount=1000, affectRows=1000 2024-10-16 13:47:13.940 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057560, nodeCount=1000, affectRows=1000 2024-10-16 13:47:13.945 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057560, nodeCount=1000, validCount=1000 2024-10-16 13:47:13.945 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057560, nodeCount=999, validCount=999 2024-10-16 13:47:13.953 ERROR 4486 --- [metric-archive-second-19,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '586774769288417280-1729057560' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:47:13.952093] [YB420AC531B8-00060B7DB46FF441-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '586774769288417280-1729057560' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:47:13.952093] [YB420AC531B8-00060B7DB46FF441-0-0] 2024-10-16 13:47:13.953 ERROR 4486 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=620709) Duplicate entry '586679390261809152-1729057560' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:47:13.952203] [YB420AC531B8-00060B7DB419652C-0-0]; nested exception is java.sql.BatchUpdateException: (conn=620709) Duplicate entry '586679390261809152-1729057560' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:47:13.952203] [YB420AC531B8-00060B7DB419652C-0-0] 2024-10-16 13:47:13.963 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057560, nodeCount=1000, affectRows=1000 2024-10-16 13:47:13.966 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057560, nodeCount=1000, validCount=1000 2024-10-16 13:47:13.994 INFO 4486 --- [prometheus-minute-schedule-1,d52615c2e846502b,558c869321cba108] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:14.000 INFO 4486 --- [prometheus-minute-schedule-1,d52615c2e846502b,558c869321cba108] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:14.070 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057560, nodeCount=194, affectRows=194 2024-10-16 13:47:14.070 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057560, nodeCount=192, validCount=192 2024-10-16 13:47:14.127 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057560, nodeCount=313, affectRows=313 2024-10-16 13:47:14.128 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057560, nodeCount=313, validCount=313 2024-10-16 13:47:14.875 INFO 4486 --- [metric-archive-async-20,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057560, nodeCount=480, affectRows=480 2024-10-16 13:47:15.277 INFO 4486 --- [ocp-async-5,03dae9d89d8a986d,991cfa33306cebd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:15.391 INFO 4486 --- [ocp-schedules-18,f6eee947750fe64a,afe75167be2122af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:47:15.423 WARN 4486 --- [scanner-13284929-1,f6eee947750fe64a,1815eac83a2b7ba8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:47:15.461 INFO 4486 --- [alarm-detect-18,f6eee947750fe64a,55458483789f209e] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057605 ~ 1729057625 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057620, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057620, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057620=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:47:15.592 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,f94acde2aa3a643e,127cc56f3b01e7d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=f94acde2aa3a643e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:47:15.593 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,f94acde2aa3a643e,127cc56f3b01e7d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=f94acde2aa3a643e, duration=3 ms] 2024-10-16 13:47:15.663 INFO 4486 --- [metric-archive-async-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057560, nodeCount=480, validCount=480 2024-10-16 13:47:16.098 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:16.994 INFO 4486 --- [prometheus-minute-schedule-1,81f4481982b2ba1c,3985bac4899ce4dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:17.001 INFO 4486 --- [prometheus-minute-schedule-1,81f4481982b2ba1c,3985bac4899ce4dc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:17.070 INFO 4486 --- [ocp-schedules-12,ac0fd78aff4fa1c3,608c0591766b0407] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:47:17.177 INFO 4486 --- [ocp-schedules-8,4bfe8a4765e80e1a,bbb8672964d41755] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:47:17.182 INFO 4486 --- [ocp-schedules-29,96e7960747b0289c,c114afe96b2176a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:47:17.277 INFO 4486 --- [ocp-async-6,e4ba62cdd7331d25,0badb41f40ec7ffe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:17.689 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,068be33fdbcb8609,ab24fb2a59b0767d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=068be33fdbcb8609, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:17.689 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,068be33fdbcb8609,ab24fb2a59b0767d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=068be33fdbcb8609, duration=1 ms] 2024-10-16 13:47:19.054 INFO 4486 --- [ocp-schedules-6,5ee4a08b9ab539a2,d39d28c434f1e3de] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:47:19.099 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:19.277 INFO 4486 --- [ocp-async-8,8c12e9a274d7a3dd,e567223eeadfadb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:19.643 INFO 4486 --- [ocp-schedules-16,e94fdbba3d7be92a,59df651991b16cf1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:47:19.644 INFO 4486 --- [ocp-schedules-16,9270eabd0e7b367f,ed314c13804a0bdf] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:47:19.646 INFO 4486 --- [ocp-schedules-16,9270eabd0e7b367f,ed314c13804a0bdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:47:19.677 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.678 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:19.682 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.682 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:19.709 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.710 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:19.710 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:19.716 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.716 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:47:19.751 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:19.752 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.753 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:19.753 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:47:19.761 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.764 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:47:19.766 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:19.780 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:19.780 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:19.795 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:19.804 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:19.805 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.806 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:19.807 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:47:19.812 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:47:19.816 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:19.818 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:19.820 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,03fa2a820b504578,5d48c07805e6bd7c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=03fa2a820b504578, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:19.820 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,03fa2a820b504578,5d48c07805e6bd7c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=03fa2a820b504578, duration=1 ms] 2024-10-16 13:47:19.829 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:19.831 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.832 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:47:19.842 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:47:19.863 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:47:19.863 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:47:19.885 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:19.886 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.887 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:19.891 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.894 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:47:19.897 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:19.917 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:47:19.925 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:19.925 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:19.926 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:19.931 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:19.932 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:47:19.941 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:47:19.941 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:47:19.954 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:19.954 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:19.984 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:19.984 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:19.994 INFO 4486 --- [prometheus-minute-schedule-1,9969d76e1a96df9f,1d2ad72a44f1025a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:20.002 INFO 4486 --- [prometheus-minute-schedule-1,9969d76e1a96df9f,1d2ad72a44f1025a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:20.018 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:20.018 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:20.045 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:20.045 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:20.072 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:20.097 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:20.098 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:20.099 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:20.104 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:47:20.107 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:47:20.111 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:47:20.113 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:47:20.134 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:47:20.280 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:47:20.307 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:20.309 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:20.310 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:47:20.343 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:47:20.343 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:47:20.790 INFO 4486 --- [ocp-schedules-2,aaf99cc2de90ba81,65da2a6c1ad475be] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:47:20.869 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 13:47:20.869 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 13:47:20.996 INFO 4486 --- [prometheus-cache-monitor1,6a53f152f40e3e72,0bfbdfb6333632ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13860,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:47:20.996 INFO 4486 --- [prometheus-cache-monitor1,6a53f152f40e3e72,0bfbdfb6333632ef] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:47:21.277 INFO 4486 --- [ocp-async-4,0a0d9bb8a3670f46,ad2478ed26005f3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:21.444 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,59c95cfbbdc71658,9126b7f542831826] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=59c95cfbbdc71658, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:21.445 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,59c95cfbbdc71658,9126b7f542831826] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=59c95cfbbdc71658, duration=2 ms] 2024-10-16 13:47:21.759 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e27190eaf9093b40,3c4e9de89f3d3453] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e27190eaf9093b40, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:21.759 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e27190eaf9093b40,3c4e9de89f3d3453] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e27190eaf9093b40, duration=1 ms] 2024-10-16 13:47:22.099 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:22.994 INFO 4486 --- [prometheus-minute-schedule-1,eb98bcd37b2140c9,a90cd56499a0bcd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:23.001 INFO 4486 --- [prometheus-minute-schedule-1,eb98bcd37b2140c9,a90cd56499a0bcd5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:23.007 INFO 4486 --- [ocp-async-3,7f8b69400f151c19,acbb61a1ceb166cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:47:23.278 INFO 4486 --- [ocp-async-7,9ed7633d3ad946b6,3418921ba635edcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:23.419 INFO 4486 --- [ocp-schedules-5,5c6dec3abfb37d2a,029ec78ec60790b2] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:47:23.532 INFO 4486 --- [ocp-schedules-7,33d3ffb50555ed79,0b1b2ef6443ad44a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:47:24.731 INFO 4486 --- [ocp-schedules-18,b18fa07fa2b513b5,4f716e94bc0e47d0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:47:25.100 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:25.278 INFO 4486 --- [ocp-async-9,c1d8a0b30058a676,a4a6443d69cf171f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:25.556 INFO 4486 --- [ocp-schedules-14,f003f9adee542f6b,4ca0e611d5210736] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:47:25.587 WARN 4486 --- [scanner-13285137-1,f003f9adee542f6b,8e9f606f450d2177] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:47:25.588 INFO 4486 --- [ocp-schedules-27,46c4580c1e1d3daf,000c7c7702b7c888] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:47:25.588 INFO 4486 --- [ocp-schedules-27,46c4580c1e1d3daf,000c7c7702b7c888] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:47:25.638 INFO 4486 --- [Collect all tenants major compactions2,46c4580c1e1d3daf,550b7a062d87bc9d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:47:25.639 INFO 4486 --- [Collect all tenants major compactions1,46c4580c1e1d3daf,1b63ffc93800ed3d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:47:25.650 INFO 4486 --- [Collect all tenants major compactions3,46c4580c1e1d3daf,d0a8e258e8e0fc77] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:47:25.658 INFO 4486 --- [Collect all tenants major compactions4,46c4580c1e1d3daf,e481cdb27a37464c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:47:25.663 INFO 4486 --- [alarm-detect-12,f003f9adee542f6b,6a137709c608f5e1] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057620 ~ 1729057635 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057635, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057635, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057635=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:47:25.665 INFO 4486 --- [Collect all tenants major compactions5,46c4580c1e1d3daf,2044889b342f71b0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:47:25.687 INFO 4486 --- [Collect all tenants major compactions2,46c4580c1e1d3daf,550b7a062d87bc9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:25.687 INFO 4486 --- [Collect all tenants major compactions1,46c4580c1e1d3daf,1b63ffc93800ed3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:25.688 INFO 4486 --- [Collect all tenants major compactions2,46c4580c1e1d3daf,550b7a062d87bc9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:47:25.689 INFO 4486 --- [Collect all tenants major compactions1,46c4580c1e1d3daf,1b63ffc93800ed3d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:47:25.692 INFO 4486 --- [Collect all tenants major compactions2,46c4580c1e1d3daf,550b7a062d87bc9d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:47:25.692 INFO 4486 --- [Collect all tenants major compactions3,46c4580c1e1d3daf,d0a8e258e8e0fc77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:25.692 INFO 4486 --- [Collect all tenants major compactions1,46c4580c1e1d3daf,1b63ffc93800ed3d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:47:25.695 INFO 4486 --- [Collect all tenants major compactions3,46c4580c1e1d3daf,d0a8e258e8e0fc77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:47:25.698 INFO 4486 --- [Collect all tenants major compactions2,46c4580c1e1d3daf,550b7a062d87bc9d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:47:25.699 INFO 4486 --- [Collect all tenants major compactions3,46c4580c1e1d3daf,d0a8e258e8e0fc77] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:47:25.699 INFO 4486 --- [Collect all tenants major compactions3,46c4580c1e1d3daf,d0a8e258e8e0fc77] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:47:25.699 INFO 4486 --- [Collect all tenants major compactions1,46c4580c1e1d3daf,1b63ffc93800ed3d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:47:25.716 INFO 4486 --- [Collect all tenants major compactions4,46c4580c1e1d3daf,e481cdb27a37464c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:25.717 INFO 4486 --- [Collect all tenants major compactions5,46c4580c1e1d3daf,2044889b342f71b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:25.719 INFO 4486 --- [Collect all tenants major compactions5,46c4580c1e1d3daf,2044889b342f71b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:47:25.720 INFO 4486 --- [Collect all tenants major compactions4,46c4580c1e1d3daf,e481cdb27a37464c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:47:25.724 INFO 4486 --- [Collect all tenants major compactions5,46c4580c1e1d3daf,2044889b342f71b0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:47:25.724 INFO 4486 --- [Collect all tenants major compactions4,46c4580c1e1d3daf,e481cdb27a37464c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:47:25.724 INFO 4486 --- [Collect all tenants major compactions5,46c4580c1e1d3daf,2044889b342f71b0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:47:25.724 INFO 4486 --- [Collect all tenants major compactions4,46c4580c1e1d3daf,e481cdb27a37464c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:47:25.726 INFO 4486 --- [ocp-schedules-27,46c4580c1e1d3daf,000c7c7702b7c888] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:47:25.994 INFO 4486 --- [prometheus-minute-schedule-1,aee87c2630345ec7,909fbd7751925a2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:26.001 INFO 4486 --- [prometheus-minute-schedule-1,aee87c2630345ec7,909fbd7751925a2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:26.474 INFO 4486 --- [ocp-schedules-1,651f5b29e1969901,ea33b1c637fb8b1e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:47:27.278 INFO 4486 --- [ocp-async-4,4ad9adaf1be94847,950862a5ae0220a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:27.384 INFO 4486 --- [ocp-schedules-28,43909d4ec992d723,0dcbedac2913b1a0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:47:27.384+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:47:27.385 INFO 4486 --- [ocp-schedules-28,43909d4ec992d723,0dcbedac2913b1a0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:47:27.386 INFO 4486 --- [ocp-schedules-28,43909d4ec992d723,0dcbedac2913b1a0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:47:27.387 INFO 4486 --- [ocp-schedules-28,43909d4ec992d723,0dcbedac2913b1a0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:47:27.697 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,312ed282b4d43e03,7a831b3a88ac7643] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=312ed282b4d43e03, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:27.697 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,312ed282b4d43e03,7a831b3a88ac7643] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=312ed282b4d43e03, duration=2 ms] 2024-10-16 13:47:28.101 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:28.994 INFO 4486 --- [prometheus-minute-schedule-1,fc9a044c3812bca1,58d330559f61bb54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:29.001 INFO 4486 --- [prometheus-minute-schedule-1,fc9a044c3812bca1,58d330559f61bb54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:29.278 INFO 4486 --- [ocp-async-1,f56a0e9e980cd2c4,bccd018a4403e04d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:29.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,7e0a7e7e28dc6778,7e39e32e00d9cc3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7e0a7e7e28dc6778, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:29.827 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,7e0a7e7e28dc6778,7e39e32e00d9cc3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7e0a7e7e28dc6778, duration=1 ms] 2024-10-16 13:47:30.103 INFO 4486 --- [prometheus-second-schedule-1,d8d55c0f0e66409f,32cec544929fa840] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:47:30.353 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,b97a9d2713f3a271,02b09173064470e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=b97a9d2713f3a271, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:47:30.354 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,b97a9d2713f3a271,02b09173064470e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=b97a9d2713f3a271, duration=2 ms] 2024-10-16 13:47:30.795 INFO 4486 --- [ocp-schedules-9,6fcd973d15aef26d,d27acdda4d806f4c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:47:31.102 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:31.278 INFO 4486 --- [ocp-async-2,8ea7485e1d214a09,e44e6024d3e44d2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:31.449 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ceb3657f881f1b96,7aba175dcbc340f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ceb3657f881f1b96, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:31.449 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ceb3657f881f1b96,7aba175dcbc340f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ceb3657f881f1b96, duration=1 ms] 2024-10-16 13:47:31.763 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e824ad18ee4240af,7940809b28f401d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e824ad18ee4240af, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:31.763 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e824ad18ee4240af,7940809b28f401d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e824ad18ee4240af, duration=1 ms] 2024-10-16 13:47:31.994 INFO 4486 --- [prometheus-minute-schedule-1,829a06eb3ca09c9a,a00462abe6cd1a6c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:32.001 INFO 4486 --- [prometheus-minute-schedule-1,829a06eb3ca09c9a,a00462abe6cd1a6c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:33.007 INFO 4486 --- [ocp-async-8,ce3157cea18f2045,a15748e68caac564] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:47:33.279 INFO 4486 --- [ocp-async-3,afd048c7d5517bf7,3ee2bfad693ca891] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:34.102 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:34.994 INFO 4486 --- [prometheus-minute-schedule-1,379ef8887fc416f1,5d4b44d8e78df311] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:34.999 INFO 4486 --- [prometheus-minute-schedule-1,379ef8887fc416f1,5d4b44d8e78df311] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:35.279 INFO 4486 --- [ocp-async-6,70259a1ac36029d2,4833cc729d94d59b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:35.764 INFO 4486 --- [ocp-schedules-28,a4467024e6e479e4,55eb0698874c74e6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:47:35.776 WARN 4486 --- [scanner-13285458-1,a4467024e6e479e4,73208e1ae0a3e7c8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:47:37.063 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:47:37.064 INFO 4486 --- [ocp-schedules-14,dafa6252ddf69c39,00b5500a2a9ed61f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:47:37.066 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:47:37.067 INFO 4486 --- [ocp-schedules-12,1134b40f9c3ff450,ff22686946f32145] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:47:37.068 INFO 4486 --- [ocp-schedules-12,1134b40f9c3ff450,ff22686946f32145] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:47:37.070 INFO 4486 --- [ocp-schedules-14,dafa6252ddf69c39,00b5500a2a9ed61f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:47:37.070 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:47:37.081 INFO 4486 --- [ocp-schedules-28,ee4bfc7a09f4e226,57dba4a3a2e2c68b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:47:37.083 INFO 4486 --- [ocp-schedules-28,ee4bfc7a09f4e226,57dba4a3a2e2c68b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:47:37.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:47:37.090 INFO 4486 --- [ocp-schedules-23,108545f7bb7a0020,10702210416ec702] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:47:37.090 INFO 4486 --- [ocp-schedules-19,bde10948ac1dd324,0475bd1db5c2bace] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:47:37.090 INFO 4486 --- [ocp-schedules-15,bef8b18e587f1ed8,819277279db5fa4c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:47:37.090 INFO 4486 --- [host-check-agent-status1,108545f7bb7a0020,a594f5c09ed6a356] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:47:37.091 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:47:37.091 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:47:37.091 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:47:37.092 INFO 4486 --- [host-check-agent-status2,108545f7bb7a0020,19c5348f5a437154] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:47:37.092 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:47:37.092 INFO 4486 --- [host-check-agent-status3,108545f7bb7a0020,acf6f66fb96129a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:47:37.093 INFO 4486 --- [host-check-clockdiff2,bef8b18e587f1ed8,247c42c82d8b077a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:47:37.093 INFO 4486 --- [host-check-agent-status4,108545f7bb7a0020,c3e35ddea7243d11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:47:37.093 INFO 4486 --- [host-check-clockdiff3,bef8b18e587f1ed8,e677d0778a445c82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:47:37.093 INFO 4486 --- [host-check-clockdiff1,bef8b18e587f1ed8,39f0b981eb68fed4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:47:37.093 INFO 4486 --- [host-check-clockdiff4,bef8b18e587f1ed8,43f92f6e20f77aed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:47:37.106 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:37.110 INFO 4486 --- [host-check-agent-status2,108545f7bb7a0020,19c5348f5a437154] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:37.111 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:37.136 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:47:37.136 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:37.138 INFO 4486 --- [host-check-clockdiff1,bef8b18e587f1ed8,39f0b981eb68fed4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057657 0 0 , error= 2024-10-16 13:47:37.138 INFO 4486 --- [host-check-clockdiff2,bef8b18e587f1ed8,247c42c82d8b077a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057657 4 4 , error= 2024-10-16 13:47:37.139 INFO 4486 --- [host-check-agent-status1,108545f7bb7a0020,a594f5c09ed6a356] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:37.139 INFO 4486 --- [host-check-agent-status4,108545f7bb7a0020,c3e35ddea7243d11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:47:37.139 INFO 4486 --- [host-check-agent-status3,108545f7bb7a0020,acf6f66fb96129a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:37.139 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:37.139 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:37.140 INFO 4486 --- [host-check-agent-status2,108545f7bb7a0020,19c5348f5a437154] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:37.140 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:47:37.141 INFO 4486 --- [host-check-agent-status1,108545f7bb7a0020,a594f5c09ed6a356] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:37.141 INFO 4486 --- [host-check-agent-status3,108545f7bb7a0020,acf6f66fb96129a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:37.141 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:37.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.141 INFO 4486 --- [host-check-agent-status4,108545f7bb7a0020,c3e35ddea7243d11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:47:37.141 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:37.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:37.142 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:47:37.145 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,eea4ff69673ee61f,cadf445d53bcf091] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=eea4ff69673ee61f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:47:37.145 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,eea4ff69673ee61f,cadf445d53bcf091] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=eea4ff69673ee61f, duration=1 ms] 2024-10-16 13:47:37.164 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:47:37.164 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:47:37.165 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:47:37.166 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:47:37.166 INFO 4486 --- [host-check-agent-status2,108545f7bb7a0020,19c5348f5a437154] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:47:37.167 INFO 4486 --- [host-check-agent-status1,108545f7bb7a0020,a594f5c09ed6a356] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:47:37.167 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:47:37.170500950+08:00 2024-10-16 13:47:37.167 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:47:37.170563748+08:00 2024-10-16 13:47:37.170 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:37.170 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:47:37.167472029+08:00 2024-10-16 13:47:37.171 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:47:37.167+08:00 2024-10-16 13:47:37.171 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:47:37.167+08:00 2024-10-16 13:47:37.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.172 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:47:37.170+08:00 2024-10-16 13:47:37.172 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:47:37.175596661+08:00 2024-10-16 13:47:37.172 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:37.173 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:47:37.172+08:00 2024-10-16 13:47:37.175 INFO 4486 --- [host-check-agent-status2,108545f7bb7a0020,19c5348f5a437154] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:47:37.175 INFO 4486 --- [host-check-agent-status2,108545f7bb7a0020,19c5348f5a437154] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:47:37.175 INFO 4486 --- [host-check-agent-status2,108545f7bb7a0020,19c5348f5a437154] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:47:37.176 INFO 4486 --- [host-check-agent-status3,108545f7bb7a0020,acf6f66fb96129a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:47:37.176 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:47:37.176 INFO 4486 --- [host-check-agent-status1,108545f7bb7a0020,a594f5c09ed6a356] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:47:37.176 INFO 4486 --- [host-check-agent-status1,108545f7bb7a0020,a594f5c09ed6a356] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:47:37.176 INFO 4486 --- [host-check-agent-status1,108545f7bb7a0020,a594f5c09ed6a356] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:47:37.176 INFO 4486 --- [host-check-agent-status4,108545f7bb7a0020,c3e35ddea7243d11] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:47:37.178 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:47:37.167+08:00 and status AVAILABLE 2024-10-16 13:47:37.178 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.180 INFO 4486 --- [host-check-agent-status3,108545f7bb7a0020,acf6f66fb96129a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:47:37.180 INFO 4486 --- [host-check-agent-status3,108545f7bb7a0020,acf6f66fb96129a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:47:37.180 INFO 4486 --- [host-check-agent-status3,108545f7bb7a0020,acf6f66fb96129a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:47:37.180 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:47:37.181 INFO 4486 --- [host-check-agent-status4,108545f7bb7a0020,c3e35ddea7243d11] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:47:37.181 INFO 4486 --- [host-check-agent-status4,108545f7bb7a0020,c3e35ddea7243d11] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:47:37.181 INFO 4486 --- [host-check-agent-status4,108545f7bb7a0020,c3e35ddea7243d11] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:47:37.183 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:47:37.184 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:47:37.167+08:00 and status AVAILABLE 2024-10-16 13:47:37.184 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.185 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:47:37.170+08:00 and status AVAILABLE 2024-10-16 13:47:37.185 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:47:37.186 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:47:37.187 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:47:37.172+08:00 and status AVAILABLE 2024-10-16 13:47:37.187 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:47:37.187 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.188 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:37.188 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.189 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:47:37.189 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:47:37.189 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:37.195 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.195 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:47:37.196 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:47:37.196 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:37.197 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:47:37.197 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:37.202 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:37.202 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.202 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.202 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.202 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.202 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.203 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:37.203 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.210 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.211 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:47:37.215 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:37.222 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:47:37.224 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:37.224 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:37.224 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:47:37.226 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.227 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:47:37.229 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.230 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:47:37.240 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.241 INFO 4486 --- [ocp-schedules-4,8211954e8fd2c067,49fc4d30b378af4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:47:37.245 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:47:37.255 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:47:37.256 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:47:37.260 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:47:37.268 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:47:37.272 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:47:37.278 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:47:37.279 INFO 4486 --- [ocp-async-2,4d8eeab45c065625,b31699ca86cdfe5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:37.279 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:37.286 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:37.288 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:37.289 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:47:37.303 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:47:37.308 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:47:37.317 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:47:37.318 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:47:37.327 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:47:37.332 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:47:37.334 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:37.338 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:47:37.353 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:37.355 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:37.360 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:47:37.362 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:47:37.371 INFO 4486 --- [host-check-host-status3,bde10948ac1dd324,1402ca2ccc33dfdd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:47:37.373 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:47:37.389 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:47:37.390 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:47:37.393 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:47:37.401 INFO 4486 --- [host-check-host-status2,bde10948ac1dd324,e77f4b80ea672668] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:47:37.403 INFO 4486 --- [host-check-host-status4,bde10948ac1dd324,e1e4edd5f0d32fce] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:47:37.438 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:47:37.447 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:37.449 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.449 INFO 4486 --- [host-check-host-status1,bde10948ac1dd324,231d4960b8548dfb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:47:37.450 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:47:37.451 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.451 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.453 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:47:37.464 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.464 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.464 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.464 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.464 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.465 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.493 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:47:37.525 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:47:37.565 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:37.581 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:37.582 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.583 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:47:37.584 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:47:37.588 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.589 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:47:37.590 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.590 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.591 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:47:37.601 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.601 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.601 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.601 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.601 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.602 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.609 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:47:37.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b2f8717c7a97c8f6,7be093e7a7da1be0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b2f8717c7a97c8f6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:37.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b2f8717c7a97c8f6,7be093e7a7da1be0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b2f8717c7a97c8f6, duration=1 ms] 2024-10-16 13:47:37.797 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:47:37.846 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:37.848 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.849 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:47:37.850 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.851 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.852 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:47:37.861 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.861 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.861 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.861 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.861 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:37.862 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:37.879 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:47:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:47:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:47:37.994 INFO 4486 --- [prometheus-minute-schedule-1,0ebce66e76103e1b,5a3f62916d56fe7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:38.000 INFO 4486 --- [prometheus-minute-schedule-1,0ebce66e76103e1b,5a3f62916d56fe7a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:38.058 INFO 4486 --- [host-check-clockdiff4,bef8b18e587f1ed8,43f92f6e20f77aed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057658 4 4 , error= 2024-10-16 13:47:38.135 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:47:38.182 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:38.185 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:38.186 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:47:38.187 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:38.188 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:38.189 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:47:38.199 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:38.200 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:38.200 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:38.200 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:38.200 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:47:38.200 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:38.277 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:47:38.446 INFO 4486 --- [ocp-schedules-13,34c3ef628e0b941f,ff3d19ce3d899ff3] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:47:38.446 INFO 4486 --- [ocp-schedules-13,34c3ef628e0b941f,ff3d19ce3d899ff3] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:47:38.447 INFO 4486 --- [ocp-schedules-13,34c3ef628e0b941f,ff3d19ce3d899ff3] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:47:38.473 INFO 4486 --- [ocp-schedules-4,a35f4b46a07f851b,49c59c111e8d3ae5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13860,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:47:38.473 INFO 4486 --- [ocp-schedules-4,a35f4b46a07f851b,49c59c111e8d3ae5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:47:38.511 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:47:38.280911, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:47:38.511 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:47:38.280911, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:47:38.594 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:47:38.280911} } 2024-10-16 13:47:38.660 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:47:38.667 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:47:38.671 INFO 4486 --- [ocp-schedules-7,7bbc5e96923236fc,71f54bb7bc049118] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:47:39.059 INFO 4486 --- [ocp-schedules-17,be3be4dc671c4f9a,eb7cc2ca138ae402] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:47:39.279 INFO 4486 --- [ocp-async-4,5d6b643064244b33,904ba6b92cd6dc96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:39.638 INFO 4486 --- [host-check-clockdiff3,bef8b18e587f1ed8,e677d0778a445c82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057659 4 4 , error= 2024-10-16 13:47:39.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,66fc0f8442df0243,53002c4d9c94351b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=66fc0f8442df0243, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:39.832 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,66fc0f8442df0243,53002c4d9c94351b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=66fc0f8442df0243, duration=1 ms] 2024-10-16 13:47:40.007 INFO 4486 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:47:40.007 INFO 4486 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:47:40.027 INFO 4486 --- [Check obproxy2,a4c5d48c74861ca6,a086831902f1713a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:47:40.027 INFO 4486 --- [Check obproxy1,a9cbdf8d76a56947,7c904dfe1c9af8e4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:47:40.027 INFO 4486 --- [Check obproxy1,a9cbdf8d76a56947,7c904dfe1c9af8e4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:40.027 INFO 4486 --- [Check obproxy2,a4c5d48c74861ca6,a086831902f1713a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:40.032 INFO 4486 --- [Collect obproxy1,b0571a4c207ddd23,328a10693ca7266b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:47:40.032 INFO 4486 --- [Collect obproxy2,301a62a0fc1f402f,12de04158533ad8f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:47:40.032 INFO 4486 --- [Collect obproxy1,b0571a4c207ddd23,328a10693ca7266b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:40.032 INFO 4486 --- [Collect obproxy2,301a62a0fc1f402f,12de04158533ad8f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:47:40.054 INFO 4486 --- [Check obproxy2,a4c5d48c74861ca6,a086831902f1713a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:47:40.054 INFO 4486 --- [Check obproxy1,a9cbdf8d76a56947,7c904dfe1c9af8e4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:47:40.054 INFO 4486 --- [Check obproxy2,a4c5d48c74861ca6,a086831902f1713a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:47:40.054 INFO 4486 --- [Check obproxy1,a9cbdf8d76a56947,7c904dfe1c9af8e4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:47:40.055 INFO 4486 --- [Collect obproxy2,301a62a0fc1f402f,12de04158533ad8f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:47:40.061 INFO 4486 --- [Collect obproxy2,301a62a0fc1f402f,12de04158533ad8f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:47:40.061 INFO 4486 --- [Collect obproxy1,b0571a4c207ddd23,328a10693ca7266b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:47:40.066 INFO 4486 --- [Collect obproxy2,301a62a0fc1f402f,12de04158533ad8f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:47:40.066 INFO 4486 --- [Collect obproxy1,b0571a4c207ddd23,328a10693ca7266b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:47:40.071 INFO 4486 --- [Collect obproxy1,b0571a4c207ddd23,328a10693ca7266b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:47:40.138 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:40.801 INFO 4486 --- [ocp-schedules-21,812bf590ec4513bc,1b9d74d21b0e74fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:47:40.994 INFO 4486 --- [prometheus-minute-schedule-1,8829a9033f2c1507,b5bd34c1a3cbf4af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:40.995 INFO 4486 --- [prometheus-hour-schedule-1,494df131b58121c7,3a495e8ba77fcf04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:47:41.001 INFO 4486 --- [prometheus-minute-schedule-1,8829a9033f2c1507,b5bd34c1a3cbf4af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:41.280 INFO 4486 --- [ocp-async-1,bb74b4b95059985c,faf07322e01e1093] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:41.455 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9d0475f12d105971,63e8dcc597e95cc6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9d0475f12d105971, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:41.455 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9d0475f12d105971,63e8dcc597e95cc6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9d0475f12d105971, duration=1 ms] 2024-10-16 13:47:41.767 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f336d322b095e93f,deeff1145681cba4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f336d322b095e93f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:41.767 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f336d322b095e93f,deeff1145681cba4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f336d322b095e93f, duration=1 ms] 2024-10-16 13:47:42.131 INFO 4486 --- [ocp-schedules-25,e52ee67f265faac6,096a9445ab1a4def] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:47:43.008 INFO 4486 --- [ocp-async-1,45e1578902d0cd26,8e38d1d3bab342ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:47:43.139 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:43.280 INFO 4486 --- [ocp-async-9,3874cf7bc2653d6a,6e5ad85f32a526b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:43.994 INFO 4486 --- [prometheus-minute-schedule-1,015295a6a7403014,5bedb3a33c711b93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:44.000 INFO 4486 --- [prometheus-minute-schedule-1,015295a6a7403014,5bedb3a33c711b93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:45.280 INFO 4486 --- [ocp-async-3,ebd3bd16e6ef6370,a7b67b8a2dfd0c81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:45.520 INFO 4486 --- [ocp-schedules-9,11637ee87f7a6683,7114e57d54ef1046] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:47:45.525 INFO 4486 --- [ocp-schedules-9,11637ee87f7a6683,7114e57d54ef1046] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:47:45.525 INFO 4486 --- [ocp-schedules-9,11637ee87f7a6683,7114e57d54ef1046] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:47:45.542 INFO 4486 --- [ocp-schedules-15,e77b58e4a7a66e31,c55d4b606d35b6e0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:47:45.542 INFO 4486 --- [ocp-schedules-15,e77b58e4a7a66e31,c55d4b606d35b6e0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:47:45.886 INFO 4486 --- [ocp-schedules-7,7b9883af86586f0e,95ee6a2ffaf1c552] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:47:45.927 WARN 4486 --- [scanner-13285682-1,7b9883af86586f0e,c60779bae500821c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:47:46.012 INFO 4486 --- [alarm-detect-18,7b9883af86586f0e,1fd8afa2a029aa4b] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057635 ~ 1729057655 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057650, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057650, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057650=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:47:46.026 INFO 4486 --- [alarm-detect-18,7b9883af86586f0e,1fd8afa2a029aa4b] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:47:46.140 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:46.994 INFO 4486 --- [prometheus-minute-schedule-1,aaf91a6a5ee0e03c,889b28cd7dacb47d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:47.000 INFO 4486 --- [prometheus-minute-schedule-1,aaf91a6a5ee0e03c,889b28cd7dacb47d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:47.069 INFO 4486 --- [ocp-schedules-12,1134b40f9c3ff450,ff22686946f32145] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:47:47.177 INFO 4486 --- [ocp-schedules-28,ee4bfc7a09f4e226,57dba4a3a2e2c68b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:47:47.181 INFO 4486 --- [ocp-schedules-1,e304acd11ba47670,5398cad6f0237008] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:47:47.280 INFO 4486 --- [ocp-async-6,2a9e13acbfd5de38,4fd0fed584e130d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:47.706 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,db6acda7aade4a5a,7df72f5be973490e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=db6acda7aade4a5a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:47.707 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,db6acda7aade4a5a,7df72f5be973490e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=db6acda7aade4a5a, duration=2 ms] 2024-10-16 13:47:48.782 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:48.783 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:47:48.785 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:48.786 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:47:48.786 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:47:49.142 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:49.280 INFO 4486 --- [ocp-async-10,47dc073c4b48ea6e,43e26a0e4b4a1e00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:49.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e7d18d41cfb17622,825a467fc82de675] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e7d18d41cfb17622, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:49.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e7d18d41cfb17622,825a467fc82de675] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e7d18d41cfb17622, duration=1 ms] 2024-10-16 13:47:49.994 INFO 4486 --- [prometheus-minute-schedule-1,56bcece4d2da25eb,0d25eb7f4e6d24df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:50.000 INFO 4486 --- [prometheus-minute-schedule-1,56bcece4d2da25eb,0d25eb7f4e6d24df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:50.344 INFO 4486 --- [ocp-schedules-16,9270eabd0e7b367f,ed314c13804a0bdf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:47:50.344 INFO 4486 --- [ocp-schedules-16,9aa987603d9edb04,e513b0a22164b776] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:47:50.346 INFO 4486 --- [ocp-schedules-16,9aa987603d9edb04,e513b0a22164b776] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:47:50.374 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.374 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:50.383 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.384 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:50.407 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.408 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:50.409 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.410 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.410 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:47:50.438 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.439 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.440 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:50.441 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:47:50.441 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.444 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:47:50.445 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:50.454 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.454 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:50.465 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.467 INFO 4486 --- [ocp-schedules-18,d1e1c0d0efa529cc,05ad211dd9043312] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:47:50.467 INFO 4486 --- [ocp-schedules-18,d1e1c0d0efa529cc,05ad211dd9043312] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:47:50.473 INFO 4486 --- [ocp-schedules-18,d1e1c0d0efa529cc,05ad211dd9043312] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:47:50.473 INFO 4486 --- [ocp-schedules-18,d1e1c0d0efa529cc,05ad211dd9043312] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:47:50.474 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.475 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.475 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:50.476 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:47:50.477 INFO 4486 --- [ocp-schedules-18,d1e1c0d0efa529cc,05ad211dd9043312] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:47:50.477 INFO 4486 --- [ocp-schedules-18,d1e1c0d0efa529cc,05ad211dd9043312] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:47:50.482 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:47:50.486 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:50.488 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:50.506 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.508 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.509 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:47:50.509 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:47:50.546 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:47:50.547 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:47:50.560 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.562 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.563 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:47:50.566 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.569 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:47:50.572 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:50.592 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:47:50.597 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.597 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:50.602 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.603 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.604 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:47:50.614 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:47:50.614 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057086920000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:47:50.620 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.620 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:50.645 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.645 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:50.672 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.672 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:47:50.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:47:50.728 INFO 4486 --- [ocp-schedules-29,d61577fc0d1567e8,614c7a79f973a0ce] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:47:50.728 INFO 4486 --- [ocp-schedules-29,d61577fc0d1567e8,614c7a79f973a0ce] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:47:50.728 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:47:50.728 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:47:50.728 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:47:50.728 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:47:50.728 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:47:50.733 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:47:50.734 INFO 4486 --- [ocp-schedules-29,d61577fc0d1567e8,614c7a79f973a0ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:47:50.734 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:47:50.734 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:50.734 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:50.735 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:47:50.736 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:47:50.736 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:50.736 INFO 4486 --- [ocp-schedules-29,d61577fc0d1567e8,614c7a79f973a0ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:47:50.736 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:50.736 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:50.737 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:47:50.737 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:50.758 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:47:50.763 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:47:50.764 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:47:50.767 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.767 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:47:50.768 INFO 4486 --- [ocp-schedules-29,d61577fc0d1567e8,614c7a79f973a0ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:47:50.768 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:47:50.768 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.769 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:47:50.773 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:47:50.774 INFO 4486 --- [ocp-schedules-29,d61577fc0d1567e8,614c7a79f973a0ce] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:47:50.778 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:47:50.783 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:47:50.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:47:50.806 INFO 4486 --- [ocp-schedules-11,432efdf6bad0e07b,aa5b6a008bf5e150] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:47:50.806 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:47:50.940 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:47:50.966 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:47:50.968 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:47:50.969 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:47:51.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:47:51.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:47:51.281 INFO 4486 --- [ocp-async-5,f331e038b2476939,cc4dd950f22a48f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:51.461 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b80efaca4af309b7,2afaa1f9f8110b0f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b80efaca4af309b7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:51.461 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b80efaca4af309b7,2afaa1f9f8110b0f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b80efaca4af309b7, duration=2 ms] 2024-10-16 13:47:51.772 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bafb78579cf31e11,8cc0840b274b191c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bafb78579cf31e11, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:51.772 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bafb78579cf31e11,8cc0840b274b191c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bafb78579cf31e11, duration=1 ms] 2024-10-16 13:47:52.143 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:52.337 INFO 4486 --- [ocp-schedules-23,223f53e2eca1cdeb,f7659c81f194edc6] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:47:52.994 INFO 4486 --- [prometheus-minute-schedule-1,ff1dc3551758119e,b47385861085658d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:53.000 INFO 4486 --- [prometheus-minute-schedule-1,ff1dc3551758119e,b47385861085658d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:53.008 INFO 4486 --- [ocp-async-3,7c868e632bfbb11f,b8615ce404ea7ad5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:47:53.281 INFO 4486 --- [ocp-async-6,2eee1609d87650cf,70cd9309e52722a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:55.144 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:55.281 INFO 4486 --- [ocp-async-9,e708939ddab9c24a,9e42123752f81edf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:55.994 INFO 4486 --- [prometheus-minute-schedule-1,dc96ae854f096fbb,23593c219d1f20ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:56.001 INFO 4486 --- [prometheus-minute-schedule-1,dc96ae854f096fbb,23593c219d1f20ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:56.107 INFO 4486 --- [ocp-schedules-1,5f3dbfb66ed4c0d5,c473ae78358d2be5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:47:56.156 WARN 4486 --- [scanner-13286066-1,5f3dbfb66ed4c0d5,0995adc17ae7c4e0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:47:56.201 INFO 4486 --- [alarm-detect-10,5f3dbfb66ed4c0d5,7ac966e80ef96254] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057650 ~ 1729057666 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057665, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057665, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057665=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:47:56.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,fc36eeb5cdf41d1d,50393327762a0f82] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=fc36eeb5cdf41d1d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:47:56.983 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,fc36eeb5cdf41d1d,50393327762a0f82] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=fc36eeb5cdf41d1d, duration=3 ms] 2024-10-16 13:47:57.281 INFO 4486 --- [ocp-async-5,8e2c9a9e166ce9cc,5218bf5f08dc7a5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:57.742 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5a12a571cd7ac3a8,1754bb3aab84a789] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5a12a571cd7ac3a8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:57.742 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5a12a571cd7ac3a8,1754bb3aab84a789] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5a12a571cd7ac3a8, duration=1 ms] 2024-10-16 13:47:58.146 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:47:58.994 INFO 4486 --- [prometheus-minute-schedule-1,03a1404e74f0367d,4af5ad3ca203fb85] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057680, interval=60 2024-10-16 13:47:59.000 INFO 4486 --- [prometheus-minute-schedule-1,03a1404e74f0367d,4af5ad3ca203fb85] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:47:59.063 INFO 4486 --- [ocp-schedules-3,1ad50a36bce3f33e,274a64c8dde0d044] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:47:59.282 INFO 4486 --- [ocp-async-1,0b56fbb2856a720f,51fca3e940967c2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:47:59.783 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:47:59.803 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:47:59.805 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:47:59.831 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:47:59.832 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:47:59.841 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,36380cade8e451af,f6fce503290cdb4f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=36380cade8e451af, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:47:59.842 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,36380cade8e451af,f6fce503290cdb4f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=36380cade8e451af, duration=1 ms] 2024-10-16 13:47:59.855 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:47:59.856 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:47:59.881 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:47:59.883 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:47:59.906 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:47:59.907 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:47:59.932 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:47:59.933 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:47:59.954 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:47:59.955 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:47:59.975 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:47:59.976 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:47:59.996 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:47:59.998 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:48:00.021 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:48:00.023 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:48:00.045 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:00.047 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:00.069 INFO 4486 --- [ocp-schedules-20,bbe40b972f929ed6,0a4454c8f51aaf73] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:48:00.812 INFO 4486 --- [ocp-schedules-6,d0999a014fffe074,29346c60a6900f68] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:48:01.147 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:01.282 INFO 4486 --- [ocp-async-8,0985eb9d95b61d54,de2c3ba54534a452] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:01.466 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,59879194deee1896,6a92a8517389f77f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=59879194deee1896, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:01.466 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,59879194deee1896,6a92a8517389f77f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=59879194deee1896, duration=1 ms] 2024-10-16 13:48:01.776 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a09b4c6afc65d053,2d35b1648090fdc3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a09b4c6afc65d053, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:01.776 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a09b4c6afc65d053,2d35b1648090fdc3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a09b4c6afc65d053, duration=1 ms] 2024-10-16 13:48:01.921 INFO 4486 --- [ocp-schedules-5,6f41e89533440876,5e69a1d5e725eacd] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:48:01.921+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:48:01.925 INFO 4486 --- [ocp-schedules-5,6f41e89533440876,5e69a1d5e725eacd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:48:01.934 INFO 4486 --- [ocp-schedules-5,6f41e89533440876,5e69a1d5e725eacd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:48:01.945 INFO 4486 --- [ocp-schedules-5,6f41e89533440876,5e69a1d5e725eacd] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:48:01.994 INFO 4486 --- [prometheus-minute-schedule-1,ca11d19daf4b57a1,52f4debb1289168d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:02.000 INFO 4486 --- [prometheus-minute-schedule-1,ca11d19daf4b57a1,52f4debb1289168d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:03.008 INFO 4486 --- [ocp-async-10,1234376adde120cc,8d5b4fe1616e72f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:48:03.282 INFO 4486 --- [ocp-async-3,48dced60066f8e43,082db5b3b35ec5eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:04.147 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:04.994 INFO 4486 --- [prometheus-minute-schedule-1,430dd4d18d3ed315,acfa8ae9df0d4dca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:05.000 INFO 4486 --- [prometheus-minute-schedule-1,430dd4d18d3ed315,acfa8ae9df0d4dca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:05.282 INFO 4486 --- [ocp-async-7,f76a010f565a2f41,b775f00320de37a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:06.257 INFO 4486 --- [ocp-schedules-14,56bbfd39ab97416a,b0afdfcfc7f9b98b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:48:06.271 WARN 4486 --- [scanner-13286259-1,56bbfd39ab97416a,9e2232d2f1f0ebbd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:48:06.916 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f2ad911c445bf5cb,dbd4a9f42df1f3a4] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&version=2&ObClusterId=1701313637, client=10.197.49.184, traceId=f2ad911c445bf5cb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=OceanBase,1701313637,2,{"ObRegion":"OceanBase","ObCluster":"OceanBase","ObRegionId":1701313637,"ObClusterId":1701313637,"Type":"PRIMARY","timestamp":1729057686911767,"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:48:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f2ad911c445bf5cb,dbd4a9f42df1f3a4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=OceanBase, obClusterId=1701313637, rootServerJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, groupName=OceanBase, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:48:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f2ad911c445bf5cb,dbd4a9f42df1f3a4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:48:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f2ad911c445bf5cb,dbd4a9f42df1f3a4] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=OceanBase, clusterId=1701313637, affectRows=1 2024-10-16 13:48:06.923 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f2ad911c445bf5cb,dbd4a9f42df1f3a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.184, traceId=f2ad911c445bf5cb, duration=9 ms] 2024-10-16 13:48:07.066 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:48:07.066 INFO 4486 --- [ocp-schedules-12,f3da26ea5ba044b8,85842e84f6045e44] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:48:07.069 INFO 4486 --- [ocp-schedules-12,f3da26ea5ba044b8,85842e84f6045e44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:07.081 INFO 4486 --- [ocp-schedules-27,9cf0ba99d1535954,ee65d57c4bfb15cf] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:48:07.083 INFO 4486 --- [ocp-schedules-27,9cf0ba99d1535954,ee65d57c4bfb15cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:48:07.089 INFO 4486 --- [ocp-schedules-19,770b316b5908a6a2,95cd1931aaf9c769] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:48:07.089 INFO 4486 --- [ocp-schedules-3,5b4e8059f2e00dea,d7d0801c493621e2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:48:07.090 INFO 4486 --- [ocp-schedules-26,0cc0831ec112d077,9eaf30c1ec66d445] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:48:07.090 INFO 4486 --- [host-check-clockdiff2,770b316b5908a6a2,24c0bdf4ba2bb49e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:48:07.090 INFO 4486 --- [host-check-clockdiff1,770b316b5908a6a2,a566f182c858ca46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:48:07.090 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:48:07.090 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:48:07.090 INFO 4486 --- [host-check-clockdiff4,770b316b5908a6a2,734da8149a900478] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:48:07.091 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:48:07.091 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:48:07.091 INFO 4486 --- [host-check-clockdiff3,770b316b5908a6a2,e5385f44240b126e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:48:07.093 INFO 4486 --- [host-check-agent-status1,0cc0831ec112d077,0b3907880d5f4980] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:48:07.093 INFO 4486 --- [host-check-agent-status2,0cc0831ec112d077,59601fc7bc8feae3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:48:07.093 INFO 4486 --- [host-check-agent-status3,0cc0831ec112d077,2885fdfae534466d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:48:07.093 INFO 4486 --- [host-check-agent-status4,0cc0831ec112d077,b90d2d63131ea408] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:48:07.100 INFO 4486 --- [host-check-agent-status1,0cc0831ec112d077,0b3907880d5f4980] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:07.100 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:48:07.120 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:48:07.118 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:07.138 INFO 4486 --- [host-check-agent-status4,0cc0831ec112d077,b90d2d63131ea408] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:07.138 INFO 4486 --- [host-check-agent-status3,0cc0831ec112d077,2885fdfae534466d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:48:07.138 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:07.138 INFO 4486 --- [host-check-agent-status2,0cc0831ec112d077,59601fc7bc8feae3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:48:07.138 INFO 4486 --- [host-check-clockdiff1,770b316b5908a6a2,a566f182c858ca46] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057687 0 0 , error= 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-agent-status2,0cc0831ec112d077,59601fc7bc8feae3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-agent-status3,0cc0831ec112d077,2885fdfae534466d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-agent-status4,0cc0831ec112d077,b90d2d63131ea408] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-agent-status1,0cc0831ec112d077,0b3907880d5f4980] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-clockdiff2,770b316b5908a6a2,24c0bdf4ba2bb49e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057687 4 4 , error= 2024-10-16 13:48:07.142 INFO 4486 --- [host-check-clockdiff4,770b316b5908a6a2,734da8149a900478] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057687 3 3 , error= 2024-10-16 13:48:07.148 INFO 4486 --- [host-check-clockdiff3,770b316b5908a6a2,e5385f44240b126e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057687 4 4 , error= 2024-10-16 13:48:07.149 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:07.153 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.154 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:07.155 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:48:07.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:07.172 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:48:07.172 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:48:07.172 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:48:07.173 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:07.174 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.174 INFO 4486 --- [host-check-agent-status3,0cc0831ec112d077,2885fdfae534466d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:48:07.175 INFO 4486 --- [host-check-agent-status2,0cc0831ec112d077,59601fc7bc8feae3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:48:07.175 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:48:07.174997203+08:00 2024-10-16 13:48:07.175 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:48:07.178470209+08:00 2024-10-16 13:48:07.175 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:48:07.175 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:48:07.179012037+08:00 2024-10-16 13:48:07.176 INFO 4486 --- [host-check-agent-status1,0cc0831ec112d077,0b3907880d5f4980] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:48:07.176 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:48:07.176 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:48:07.175+08:00 2024-10-16 13:48:07.176 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:48:07.175+08:00 2024-10-16 13:48:07.176 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:48:07.176+08:00 2024-10-16 13:48:07.176 INFO 4486 --- [host-check-agent-status4,0cc0831ec112d077,b90d2d63131ea408] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:48:07.178 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:48:07.182200625+08:00 2024-10-16 13:48:07.179 INFO 4486 --- [host-check-agent-status3,0cc0831ec112d077,2885fdfae534466d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:48:07.179 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:48:07.176+08:00 and status AVAILABLE 2024-10-16 13:48:07.179 INFO 4486 --- [host-check-agent-status3,0cc0831ec112d077,2885fdfae534466d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:48:07.179 INFO 4486 --- [host-check-agent-status3,0cc0831ec112d077,2885fdfae534466d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:48:07.179 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:48:07.175+08:00 and status AVAILABLE 2024-10-16 13:48:07.179 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:48:07.175+08:00 and status AVAILABLE 2024-10-16 13:48:07.179 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:48:07.179+08:00 2024-10-16 13:48:07.180 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:48:07.180 INFO 4486 --- [host-check-agent-status4,0cc0831ec112d077,b90d2d63131ea408] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:48:07.180 INFO 4486 --- [host-check-agent-status4,0cc0831ec112d077,b90d2d63131ea408] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:48:07.180 INFO 4486 --- [host-check-agent-status4,0cc0831ec112d077,b90d2d63131ea408] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:48:07.181 INFO 4486 --- [host-check-agent-status1,0cc0831ec112d077,0b3907880d5f4980] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:48:07.181 INFO 4486 --- [host-check-agent-status1,0cc0831ec112d077,0b3907880d5f4980] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:48:07.181 INFO 4486 --- [host-check-agent-status1,0cc0831ec112d077,0b3907880d5f4980] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:48:07.181 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:48:07.181 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:48:07.182 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:48:07.182 INFO 4486 --- [host-check-agent-status2,0cc0831ec112d077,59601fc7bc8feae3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:48:07.182 INFO 4486 --- [host-check-agent-status2,0cc0831ec112d077,59601fc7bc8feae3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:48:07.182 INFO 4486 --- [host-check-agent-status2,0cc0831ec112d077,59601fc7bc8feae3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:48:07.183 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:48:07.179+08:00 and status AVAILABLE 2024-10-16 13:48:07.185 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:48:07.188 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.189 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:07.190 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:07.190 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:48:07.191 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:07.191 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:07.192 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:07.192 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:48:07.194 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:48:07.196 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:48:07.200 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.201 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:07.211 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.212 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:07.214 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.215 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:48:07.215 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:07.215 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:07.216 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:07.222 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:07.228 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:07.230 INFO 4486 --- [ocp-schedules-18,4bc4101d4f132bef,80966d9cc2b0d205] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:48:07.247 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:48:07.248 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:48:07.259 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:48:07.260 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:48:07.260 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:48:07.271 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:48:07.277 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:07.277 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:48:07.277 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:07.282 INFO 4486 --- [ocp-async-10,9920ccc448445c3e,260cd63ba9ff78ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:07.287 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:48:07.287 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:07.302 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:07.308 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:48:07.313 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:48:07.318 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:48:07.326 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:48:07.331 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:48:07.334 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:07.340 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:48:07.341 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:07.357 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:48:07.357 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:07.370 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:48:07.370 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:48:07.372 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:48:07.381 INFO 4486 --- [host-check-host-status3,5b4e8059f2e00dea,10ec67f51e731204] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:48:07.384 INFO 4486 --- [host-check-host-status4,5b4e8059f2e00dea,e5c2360fb512148b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:48:07.391 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:07.392 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:48:07.403 INFO 4486 --- [host-check-host-status2,5b4e8059f2e00dea,67cb628c68731d58] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:48:07.449 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:48:07.459 INFO 4486 --- [host-check-host-status1,5b4e8059f2e00dea,d9e9c5a3d4f9a42d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:48:07.751 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8d8d478db23d37a4,eacee621fa5ed9e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8d8d478db23d37a4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:07.751 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8d8d478db23d37a4,eacee621fa5ed9e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8d8d478db23d37a4, duration=1 ms] 2024-10-16 13:48:07.994 INFO 4486 --- [prometheus-minute-schedule-1,cf4278026aa333f7,77cf492fce1ebd22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:08.002 INFO 4486 --- [prometheus-minute-schedule-1,cf4278026aa333f7,77cf492fce1ebd22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:09.283 INFO 4486 --- [ocp-async-9,6285f7500cf4c840,2d7c2e17703192fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:09.845 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,79fa9a94a3121abd,83a5228db1b69659] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=79fa9a94a3121abd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:09.845 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,79fa9a94a3121abd,83a5228db1b69659] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=79fa9a94a3121abd, duration=1 ms] 2024-10-16 13:48:10.007 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:48:10.007 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:48:10.029 INFO 4486 --- [Check obproxy2,72ab6e5d085ed15e,7f05016e75f6366c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:48:10.029 INFO 4486 --- [Check obproxy2,72ab6e5d085ed15e,7f05016e75f6366c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:10.032 INFO 4486 --- [Collect obproxy2,4171a8ae0cab055e,372122181d393015] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:48:10.032 INFO 4486 --- [Collect obproxy2,4171a8ae0cab055e,372122181d393015] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:10.034 INFO 4486 --- [Check obproxy1,fde473c9772eb5d7,0021991bf5eacc36] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:48:10.034 INFO 4486 --- [Check obproxy1,fde473c9772eb5d7,0021991bf5eacc36] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:10.034 INFO 4486 --- [Collect obproxy1,a23d0eaf487e4da9,f39957dbcf80ed9e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:48:10.034 INFO 4486 --- [Collect obproxy1,a23d0eaf487e4da9,f39957dbcf80ed9e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:10.058 INFO 4486 --- [Collect obproxy2,4171a8ae0cab055e,372122181d393015] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:48:10.060 INFO 4486 --- [Check obproxy2,72ab6e5d085ed15e,7f05016e75f6366c] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:48:10.060 INFO 4486 --- [Check obproxy2,72ab6e5d085ed15e,7f05016e75f6366c] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:48:10.062 INFO 4486 --- [Check obproxy1,fde473c9772eb5d7,0021991bf5eacc36] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:48:10.062 INFO 4486 --- [Check obproxy1,fde473c9772eb5d7,0021991bf5eacc36] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:48:10.063 INFO 4486 --- [Collect obproxy2,4171a8ae0cab055e,372122181d393015] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:48:10.065 INFO 4486 --- [Collect obproxy1,a23d0eaf487e4da9,f39957dbcf80ed9e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:48:10.069 INFO 4486 --- [Collect obproxy1,a23d0eaf487e4da9,f39957dbcf80ed9e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:48:10.069 INFO 4486 --- [Collect obproxy2,4171a8ae0cab055e,372122181d393015] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:48:10.071 INFO 4486 --- [Collect obproxy1,a23d0eaf487e4da9,f39957dbcf80ed9e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:48:10.150 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:10.817 INFO 4486 --- [ocp-schedules-11,48cf7589378338b0,f1313dec721361cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:48:10.977 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:48:10.994 INFO 4486 --- [prometheus-minute-schedule-1,a8528711605d60c3,32993af8f301a537] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:11.000 INFO 4486 --- [prometheus-minute-schedule-1,a8528711605d60c3,32993af8f301a537] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:11.007 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:48:11.283 INFO 4486 --- [ocp-async-3,d005b2a88762dd70,5fe94101fc4719be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:11.471 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,30d63595e7e71b2f,2de746bb982954ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=30d63595e7e71b2f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:11.471 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,30d63595e7e71b2f,2de746bb982954ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=30d63595e7e71b2f, duration=1 ms] 2024-10-16 13:48:11.780 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d368a291116fc36d,72527bfaef03670c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d368a291116fc36d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:11.780 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d368a291116fc36d,72527bfaef03670c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d368a291116fc36d, duration=1 ms] 2024-10-16 13:48:13.008 INFO 4486 --- [ocp-async-1,4152906d637f5d24,8cd1fabef745a116] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:48:13.151 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:13.283 INFO 4486 --- [ocp-async-6,d7cb3dfb5604dc82,2639ef43cf436834] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:13.994 INFO 4486 --- [prometheus-minute-schedule-1,d425cd2a5739942c,ede13eb121d4e9f1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:14.001 INFO 4486 --- [prometheus-minute-schedule-1,d425cd2a5739942c,ede13eb121d4e9f1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:14.018 ERROR 4486 --- [metric-archive-second-11,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1525855) Duplicate entry '586679390270197760-1729057620' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:48:14.016995] [YB420AC531B8-00060B7DB76885AA-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1525855) Duplicate entry '586679390270197760-1729057620' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:48:14.016995] [YB420AC531B8-00060B7DB76885AA-0-0] 2024-10-16 13:48:14.026 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057620, nodeCount=1000, affectRows=1000 2024-10-16 13:48:14.030 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057620, nodeCount=999, validCount=999 2024-10-16 13:48:14.032 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057620, nodeCount=1000, affectRows=1000 2024-10-16 13:48:14.035 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057620, nodeCount=1000, validCount=1000 2024-10-16 13:48:14.041 INFO 4486 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057620, nodeCount=1000, affectRows=1000 2024-10-16 13:48:14.046 INFO 4486 --- [metric-archive-second-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057620, nodeCount=1000, validCount=1000 2024-10-16 13:48:14.257 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057620, nodeCount=449, affectRows=449 2024-10-16 13:48:14.260 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057620, nodeCount=447, validCount=447 2024-10-16 13:48:14.848 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057620, nodeCount=126, affectRows=126 2024-10-16 13:48:14.851 INFO 4486 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057620, nodeCount=103, affectRows=103 2024-10-16 13:48:14.857 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057620, nodeCount=248, affectRows=248 2024-10-16 13:48:15.017 INFO 4486 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057620, nodeCount=103, validCount=103 2024-10-16 13:48:15.066 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057620, nodeCount=126, validCount=126 2024-10-16 13:48:15.206 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057620, nodeCount=248, validCount=248 2024-10-16 13:48:15.283 INFO 4486 --- [ocp-async-2,61fd745661ba2698,8107ddd0ffac8b1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:16.153 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:16.426 INFO 4486 --- [ocp-schedules-6,39a458ffd6fdf5d5,651458e9a4e5e968] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:48:16.438 WARN 4486 --- [scanner-13287409-1,39a458ffd6fdf5d5,08164ff222572ee1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:48:16.507 INFO 4486 --- [alarm-detect-20,39a458ffd6fdf5d5,4e0a6de256ed9f16] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057665 ~ 1729057686 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057680, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057680, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057680=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:48:16.994 INFO 4486 --- [prometheus-minute-schedule-1,31375b54f20099cb,3e8239bcfdce46c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:16.999 INFO 4486 --- [prometheus-minute-schedule-1,31375b54f20099cb,3e8239bcfdce46c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:17.070 INFO 4486 --- [ocp-schedules-12,f3da26ea5ba044b8,85842e84f6045e44] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:48:17.176 INFO 4486 --- [ocp-schedules-27,9cf0ba99d1535954,ee65d57c4bfb15cf] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:48:17.184 INFO 4486 --- [ocp-schedules-14,4f96e337d98cbff5,4b786a25609e2632] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:48:17.284 INFO 4486 --- [ocp-async-4,8d03edbb75e4a89f,026c236037473ab6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:17.756 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9871be374ebd974f,356ef088bac7b971] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9871be374ebd974f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:17.756 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9871be374ebd974f,356ef088bac7b971] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9871be374ebd974f, duration=2 ms] 2024-10-16 13:48:19.067 INFO 4486 --- [ocp-schedules-24,a378d67ef8293366,fe4279735f8b08b1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:48:19.154 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:19.284 INFO 4486 --- [ocp-async-7,b44b7b9ea84aad0b,25add1a85b0f94d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:19.928 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,39cfeb4cec04ff4d,bb363b476bbaceab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=39cfeb4cec04ff4d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:19.929 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,39cfeb4cec04ff4d,bb363b476bbaceab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=39cfeb4cec04ff4d, duration=2 ms] 2024-10-16 13:48:19.994 INFO 4486 --- [prometheus-minute-schedule-1,b77e17cf16a939ed,232a0c299a5f1832] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:20.000 INFO 4486 --- [prometheus-minute-schedule-1,b77e17cf16a939ed,232a0c299a5f1832] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:20.812 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:48:20.822 INFO 4486 --- [ocp-schedules-14,d9def8e74de6d469,da680f5b3ee345d9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:48:20.909 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a8d997887fd3b529,feee2fdb52c6b841] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=a8d997887fd3b529, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:48:20.910 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a8d997887fd3b529,feee2fdb52c6b841] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=a8d997887fd3b529, duration=2 ms] 2024-10-16 13:48:20.996 INFO 4486 --- [prometheus-cache-monitor1,897bc7e6b37ad717,d20ff0c3b523a506] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13858,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:48:20.996 INFO 4486 --- [prometheus-cache-monitor1,897bc7e6b37ad717,d20ff0c3b523a506] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:48:21.002 INFO 4486 --- [ocp-schedules-16,9aa987603d9edb04,e513b0a22164b776] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:48:21.002 INFO 4486 --- [ocp-schedules-16,76637d0aca4b6d43,9d3344338c1a2038] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:48:21.004 INFO 4486 --- [ocp-schedules-16,76637d0aca4b6d43,9d3344338c1a2038] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:21.033 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.033 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:21.048 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.049 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:21.068 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.069 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.070 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.070 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:48:21.071 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:21.098 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.099 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.099 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:21.100 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:48:21.101 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.104 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:48:21.105 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:21.115 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.115 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:21.128 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.138 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.139 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:21.140 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:48:21.146 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:48:21.150 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:21.152 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:21.173 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:48:21.199 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.201 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.202 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:48:21.223 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:48:21.223 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:48:21.247 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:48:21.254 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.255 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.255 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.256 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:21.256 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.257 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:48:21.260 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.264 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:48:21.266 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:48:21.266 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:48:21.267 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:21.284 INFO 4486 --- [ocp-async-10,d703719c4e388d6d,45643a81980fa549] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:21.294 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.294 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:21.322 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.322 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:21.348 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.349 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:21.383 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.383 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:21.411 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.411 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:21.442 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:21.469 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.471 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.472 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:21.475 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b19b827f4acbcf7c,9bc387b254b516a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b19b827f4acbcf7c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:21.476 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b19b827f4acbcf7c,9bc387b254b516a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b19b827f4acbcf7c, duration=2 ms] 2024-10-16 13:48:21.476 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:48:21.480 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:48:21.485 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:48:21.486 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:48:21.509 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:48:21.664 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:48:21.690 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:21.693 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:21.695 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:48:21.726 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:48:21.726 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:48:21.786 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4b64d5eb8d41eab0,67d52e7fcfe5eba9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4b64d5eb8d41eab0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:21.786 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4b64d5eb8d41eab0,67d52e7fcfe5eba9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4b64d5eb8d41eab0, duration=1 ms] 2024-10-16 13:48:22.155 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:22.994 INFO 4486 --- [prometheus-minute-schedule-1,3d0e069047817adf,8cd4dc1218562d61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:22.999 INFO 4486 --- [prometheus-minute-schedule-1,3d0e069047817adf,8cd4dc1218562d61] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:23.009 INFO 4486 --- [ocp-async-6,242603544306ec62,a64c3780f957bc9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:48:23.284 INFO 4486 --- [ocp-async-9,4f1d96c4826025c9,81f3401577a38c1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:23.460 INFO 4486 --- [ocp-schedules-8,292e9ae611c33d69,cffbf08deaaf45a3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:48:23.532 INFO 4486 --- [ocp-schedules-30,9fd0e971e1fec466,f7b84bdb24430827] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:48:23.534 INFO 4486 --- [ocp-schedules-30,9fd0e971e1fec466,f7b84bdb24430827] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:23.534 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:48:23.535 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:48:23.535 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:48:24.764 INFO 4486 --- [ocp-schedules-14,b9d925ce57e37d1c,5e7a5a260614f387] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:48:25.157 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:25.284 INFO 4486 --- [ocp-async-3,889d230d46b6333b,5da0443cfa4034f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:25.727 INFO 4486 --- [ocp-schedules-15,c992e19ad9015fc1,314f2a5e2fcf29ba] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:48:25.727 INFO 4486 --- [ocp-schedules-15,c992e19ad9015fc1,314f2a5e2fcf29ba] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:48:25.759 INFO 4486 --- [Collect all tenants major compactions2,c992e19ad9015fc1,e39bf1a0eac30d59] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:48:25.760 INFO 4486 --- [Collect all tenants major compactions1,c992e19ad9015fc1,d0e23ac32105ecf7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:48:25.760 INFO 4486 --- [Collect all tenants major compactions5,c992e19ad9015fc1,2b4c6e7b55259cce] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:48:25.760 INFO 4486 --- [Collect all tenants major compactions3,c992e19ad9015fc1,3c7399cf428f6d0c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:48:25.760 INFO 4486 --- [Collect all tenants major compactions4,c992e19ad9015fc1,4ee55d2099edb0d2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:48:25.773 INFO 4486 --- [Collect all tenants major compactions1,c992e19ad9015fc1,d0e23ac32105ecf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:25.773 INFO 4486 --- [Collect all tenants major compactions2,c992e19ad9015fc1,e39bf1a0eac30d59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:25.774 INFO 4486 --- [Collect all tenants major compactions1,c992e19ad9015fc1,d0e23ac32105ecf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:48:25.775 INFO 4486 --- [Collect all tenants major compactions2,c992e19ad9015fc1,e39bf1a0eac30d59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:48:25.777 INFO 4486 --- [Collect all tenants major compactions1,c992e19ad9015fc1,d0e23ac32105ecf7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:48:25.778 INFO 4486 --- [Collect all tenants major compactions1,c992e19ad9015fc1,d0e23ac32105ecf7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:48:25.778 INFO 4486 --- [Collect all tenants major compactions2,c992e19ad9015fc1,e39bf1a0eac30d59] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:48:25.778 INFO 4486 --- [Collect all tenants major compactions2,c992e19ad9015fc1,e39bf1a0eac30d59] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:48:25.780 INFO 4486 --- [Collect all tenants major compactions3,c992e19ad9015fc1,3c7399cf428f6d0c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:25.782 INFO 4486 --- [Collect all tenants major compactions3,c992e19ad9015fc1,3c7399cf428f6d0c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:48:25.788 INFO 4486 --- [Collect all tenants major compactions3,c992e19ad9015fc1,3c7399cf428f6d0c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:48:25.788 INFO 4486 --- [Collect all tenants major compactions3,c992e19ad9015fc1,3c7399cf428f6d0c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:48:25.793 INFO 4486 --- [Collect all tenants major compactions5,c992e19ad9015fc1,2b4c6e7b55259cce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:25.793 INFO 4486 --- [Collect all tenants major compactions4,c992e19ad9015fc1,4ee55d2099edb0d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:25.795 INFO 4486 --- [Collect all tenants major compactions4,c992e19ad9015fc1,4ee55d2099edb0d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:48:25.795 INFO 4486 --- [Collect all tenants major compactions5,c992e19ad9015fc1,2b4c6e7b55259cce] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:48:25.799 INFO 4486 --- [Collect all tenants major compactions4,c992e19ad9015fc1,4ee55d2099edb0d2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:48:25.799 INFO 4486 --- [Collect all tenants major compactions4,c992e19ad9015fc1,4ee55d2099edb0d2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:48:25.800 INFO 4486 --- [Collect all tenants major compactions5,c992e19ad9015fc1,2b4c6e7b55259cce] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:48:25.801 INFO 4486 --- [Collect all tenants major compactions5,c992e19ad9015fc1,2b4c6e7b55259cce] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:48:25.803 INFO 4486 --- [ocp-schedules-15,c992e19ad9015fc1,314f2a5e2fcf29ba] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:48:25.994 INFO 4486 --- [prometheus-minute-schedule-1,58e1b9bdacaf99b1,80eb47de317a4149] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:26.002 INFO 4486 --- [prometheus-minute-schedule-1,58e1b9bdacaf99b1,80eb47de317a4149] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:26.474 INFO 4486 --- [ocp-schedules-18,66f72d009f5c151a,82f97b7978000d92] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:48:26.476 INFO 4486 --- [ocp-schedules-18,66f72d009f5c151a,82f97b7978000d92] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:26.511 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:48:26.512 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.516 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:48:26.524 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:48:26.525 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.525 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.526 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:48:26.529 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:48:26.535 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.535 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:48:26.544 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.544 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:48:26.547 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.547 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:48:26.552 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.552 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:48:26.561 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.562 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:48:26.562 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:26.562 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.565 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.566 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:48:26.566 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:48:26.570 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.570 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:26.571 INFO 4486 --- [ocp-schedules-28,f6ce59e08525c436,6b72b38bab7e8c3a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:48:26.578 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.579 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:48:26.580 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.580 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.580 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:48:26.581 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:26.593 WARN 4486 --- [scanner-13287647-1,f6ce59e08525c436,d96aa1fc166b0439] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:48:26.619 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.620 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.621 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:26.621 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:48:26.621 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.622 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:48:26.675 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.677 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:48:26.685 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.686 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:48:26.690 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.691 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:48:26.692 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.693 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:48:26.701 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.702 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:48:26.712 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.713 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:48:26.716 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.717 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:26.723 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.724 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.725 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.726 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.727 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.729 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.729 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.730 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.732 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.733 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.733 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.735 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.726 INFO 4486 --- [alarm-detect-16,f6ce59e08525c436,5af5c3ec815e5c44] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057680 ~ 1729057696 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057695, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057695, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057695=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:48:26.741 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.742 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.742 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.744 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.748 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.749 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.753 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.754 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.755 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.756 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.756 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.760 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.762 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.763 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.764 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.766 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.771 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.771 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.772 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.772 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.775 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.776 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.777 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.777 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.777 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.778 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.779 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:26.779 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:48:26.780 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.783 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.785 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.789 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.791 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.795 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:48:26.900 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.900 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:48:26.901 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:26.902 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:48:26.919 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:26.923 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.923 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:26.935 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:48:26.936 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.939 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.940 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:26.940 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.941 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:48:26.941 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:48:26.941 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.943 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:48:26.943 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.944 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.947 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.948 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.950 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.951 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.953 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:26.953 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:48:26.953 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:26.953 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:48:26.953 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:26.953 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:48:26.953 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.955 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.956 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:48:26.956 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:48:26.957 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.958 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:48:26.961 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.962 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:48:26.964 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:26.965 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:48:26.967 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:48:26.970 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:48:27.026 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:27.028 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:27.029 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:48:27.038 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:27.038 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:48:27.039 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:27.039 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:48:27.039 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:48:27.039 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:48:27.043 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:48:27.043 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:48:27.285 INFO 4486 --- [ocp-async-7,9a68d1514ce7c5c0,4e00139e19aa2d9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:27.418 INFO 4486 --- [ocp-schedules-22,ed3936b7cf031d92,c17362872a27343b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:48:27.418+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:48:27.420 INFO 4486 --- [ocp-schedules-22,ed3936b7cf031d92,c17362872a27343b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:48:27.421 INFO 4486 --- [ocp-schedules-22,ed3936b7cf031d92,c17362872a27343b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:48:27.422 INFO 4486 --- [ocp-schedules-22,ed3936b7cf031d92,c17362872a27343b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:48:27.543 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,5943fc194092025b,1879dc30ebb16d9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=5943fc194092025b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:48:27.544 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,5943fc194092025b,1879dc30ebb16d9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=5943fc194092025b, duration=2 ms] 2024-10-16 13:48:27.761 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6e8297f43fc41126,9574a7d2eaa05d7f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6e8297f43fc41126, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:27.761 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6e8297f43fc41126,9574a7d2eaa05d7f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6e8297f43fc41126, duration=1 ms] 2024-10-16 13:48:28.158 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:28.822 INFO 4486 --- [ocp-schedules-22,99ae1517363a1589,7fc4cd9c6c1eff7d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:48:28.823 INFO 4486 --- [ocp-schedules-22,99ae1517363a1589,7fc4cd9c6c1eff7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:28.994 INFO 4486 --- [prometheus-minute-schedule-1,3f9781d8d3ef6169,5dc6e2c49d9ca90f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:28.999 INFO 4486 --- [prometheus-minute-schedule-1,3f9781d8d3ef6169,5dc6e2c49d9ca90f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:29.285 INFO 4486 --- [ocp-async-5,27b27f3ab3e8acf9,60e9159c18127fe8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:29.940 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c8a0966650b022fa,d1d799ddd260ce5e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c8a0966650b022fa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:29.941 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c8a0966650b022fa,d1d799ddd260ce5e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c8a0966650b022fa, duration=1 ms] 2024-10-16 13:48:30.103 INFO 4486 --- [prometheus-second-schedule-1,db0b64df65fc08c9,d48a4e976572eb32] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:48:30.827 INFO 4486 --- [ocp-schedules-2,046f8f3f5a6b5918,bedd9457795db524] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:48:31.159 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:31.285 INFO 4486 --- [ocp-async-9,02a7ec94a1b629de,9c12af0957f3d735] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:31.480 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d14be7a07bf9503f,5d754b41c72705c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d14be7a07bf9503f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:31.480 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d14be7a07bf9503f,5d754b41c72705c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d14be7a07bf9503f, duration=1 ms] 2024-10-16 13:48:31.794 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bbd5c8f770f37a27,13263f91b80848e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bbd5c8f770f37a27, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:31.794 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bbd5c8f770f37a27,13263f91b80848e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bbd5c8f770f37a27, duration=1 ms] 2024-10-16 13:48:31.994 INFO 4486 --- [prometheus-minute-schedule-1,9164b20c141c3873,88fff5af1a6153c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:32.000 INFO 4486 --- [prometheus-minute-schedule-1,9164b20c141c3873,88fff5af1a6153c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:33.009 INFO 4486 --- [ocp-async-4,8daa2b9ae054e182,1244a6076324f6cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:48:33.285 INFO 4486 --- [ocp-async-6,fcc84edd76ff5013,52a2201ca216218d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:33.536 INFO 4486 --- [ocp-schedules-30,9fd0e971e1fec466,f7b84bdb24430827] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:48:34.161 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:34.994 INFO 4486 --- [prometheus-minute-schedule-1,227b717990779cf6,15180debf9b61c62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:35.000 INFO 4486 --- [prometheus-minute-schedule-1,227b717990779cf6,15180debf9b61c62] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:35.286 INFO 4486 --- [ocp-async-2,07a573d045873941,b0891ecdbdd9e0e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:36.933 INFO 4486 --- [ocp-schedules-28,d661de99f346b215,e7c83a02f5ccbd40] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:48:36.973 WARN 4486 --- [scanner-13287973-1,d661de99f346b215,d52d41acd755969d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:48:37.065 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:48:37.068 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:48:37.068 INFO 4486 --- [ocp-schedules-2,f3c5efd0463011ba,793875d25cb250c7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:48:37.069 INFO 4486 --- [ocp-schedules-9,e88ea6c6e0ca5f06,8891cb91d1f53c24] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:48:37.077 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:48:37.078 INFO 4486 --- [ocp-schedules-2,f3c5efd0463011ba,793875d25cb250c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:37.084 INFO 4486 --- [ocp-schedules-15,bd579e56d0cd4e50,d01616fac3909620] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:48:37.086 INFO 4486 --- [ocp-schedules-15,bd579e56d0cd4e50,d01616fac3909620] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:37.089 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:48:37.093 INFO 4486 --- [ocp-schedules-9,e88ea6c6e0ca5f06,8891cb91d1f53c24] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:48:37.093 INFO 4486 --- [ocp-schedules-3,6f71ec4ceda5e8ba,2e0e18ce7dbfc66a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:48:37.093 INFO 4486 --- [ocp-schedules-4,79258b49414bfe15,53255ff76c973afb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:48:37.095 INFO 4486 --- [host-check-agent-status1,6f71ec4ceda5e8ba,1d096d93754c30c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:48:37.096 INFO 4486 --- [host-check-agent-status2,6f71ec4ceda5e8ba,83a9e0fc793ef1af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:48:37.096 INFO 4486 --- [host-check-agent-status3,6f71ec4ceda5e8ba,17934ac1cfd68782] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:48:37.096 INFO 4486 --- [host-check-clockdiff1,79258b49414bfe15,0e9c9b93f66f9c78] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:48:37.097 INFO 4486 --- [host-check-agent-status4,6f71ec4ceda5e8ba,795947a5f624b6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:48:37.100 INFO 4486 --- [ocp-schedules-7,8d1bc6b4364ff304,11b674443985f4b4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:48:37.112 INFO 4486 --- [host-check-agent-status4,6f71ec4ceda5e8ba,795947a5f624b6dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:37.112 INFO 4486 --- [host-check-agent-status3,6f71ec4ceda5e8ba,17934ac1cfd68782] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:48:37.113 INFO 4486 --- [host-check-agent-status2,6f71ec4ceda5e8ba,83a9e0fc793ef1af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:48:37.122 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:48:37.123 INFO 4486 --- [host-check-agent-status3,6f71ec4ceda5e8ba,17934ac1cfd68782] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:48:37.123 INFO 4486 --- [host-check-agent-status2,6f71ec4ceda5e8ba,83a9e0fc793ef1af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:48:37.124 INFO 4486 --- [host-check-agent-status4,6f71ec4ceda5e8ba,795947a5f624b6dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:37.128 INFO 4486 --- [host-check-clockdiff2,79258b49414bfe15,1fc8bd67966310ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:48:37.134 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:48:37.134 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:48:37.135 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:48:37.139 INFO 4486 --- [host-check-clockdiff1,79258b49414bfe15,0e9c9b93f66f9c78] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057717 0 0 , error= 2024-10-16 13:48:37.142 INFO 4486 --- [host-check-clockdiff3,79258b49414bfe15,7ec7b264aa700517] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:48:37.142 INFO 4486 --- [host-check-clockdiff4,79258b49414bfe15,de60a1cb047d47a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:48:37.142 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:37.145 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:37.152 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:48:37.175 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:48:37.176 INFO 4486 --- [host-check-agent-status1,6f71ec4ceda5e8ba,1d096d93754c30c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:37.177 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:48:37.178 INFO 4486 --- [host-check-agent-status1,6f71ec4ceda5e8ba,1d096d93754c30c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:37.182 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.183 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:37.184 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:48:37.227 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:37.229 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:48:37.235 INFO 4486 --- [host-check-clockdiff2,79258b49414bfe15,1fc8bd67966310ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057717 4 4 , error= 2024-10-16 13:48:37.235 INFO 4486 --- [host-check-clockdiff3,79258b49414bfe15,7ec7b264aa700517] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057717 4 4 , error= 2024-10-16 13:48:37.238 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:37.238 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:37.240 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:37.241 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.242 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:48:37.244 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.244 INFO 4486 --- [host-check-agent-status4,6f71ec4ceda5e8ba,795947a5f624b6dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:48:37.245 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.246 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:48:37.248 INFO 4486 --- [host-check-agent-status3,6f71ec4ceda5e8ba,17934ac1cfd68782] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:48:37.249 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:48:37.249 INFO 4486 --- [host-check-agent-status2,6f71ec4ceda5e8ba,83a9e0fc793ef1af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:48:37.250 INFO 4486 --- [host-check-agent-status1,6f71ec4ceda5e8ba,1d096d93754c30c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:48:37.251 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:48:37.251016959+08:00 2024-10-16 13:48:37.251 INFO 4486 --- [host-check-agent-status4,6f71ec4ceda5e8ba,795947a5f624b6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:48:37.251 INFO 4486 --- [host-check-agent-status4,6f71ec4ceda5e8ba,795947a5f624b6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:48:37.251 INFO 4486 --- [host-check-agent-status4,6f71ec4ceda5e8ba,795947a5f624b6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:48:37.252 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:48:37.251+08:00 2024-10-16 13:48:37.254 INFO 4486 --- [host-check-agent-status3,6f71ec4ceda5e8ba,17934ac1cfd68782] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:48:37.254 INFO 4486 --- [host-check-agent-status3,6f71ec4ceda5e8ba,17934ac1cfd68782] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:48:37.254 INFO 4486 --- [host-check-agent-status3,6f71ec4ceda5e8ba,17934ac1cfd68782] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:48:37.255 INFO 4486 --- [host-check-agent-status1,6f71ec4ceda5e8ba,1d096d93754c30c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:48:37.255 INFO 4486 --- [host-check-agent-status1,6f71ec4ceda5e8ba,1d096d93754c30c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:48:37.255 INFO 4486 --- [host-check-agent-status1,6f71ec4ceda5e8ba,1d096d93754c30c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:48:37.255 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:48:37.256 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:48:37.251+08:00 and status AVAILABLE 2024-10-16 13:48:37.256 INFO 4486 --- [host-check-agent-status2,6f71ec4ceda5e8ba,83a9e0fc793ef1af] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:48:37.256 INFO 4486 --- [host-check-agent-status2,6f71ec4ceda5e8ba,83a9e0fc793ef1af] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:48:37.256 INFO 4486 --- [host-check-agent-status2,6f71ec4ceda5e8ba,83a9e0fc793ef1af] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:48:37.257 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:48:37.260711838+08:00 2024-10-16 13:48:37.257 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:48:37.257+08:00 2024-10-16 13:48:37.257 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.258 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:37.259 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.259 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:48:37.259 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.259 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.259 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.259 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.260 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.260 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:48:37.257+08:00 and status AVAILABLE 2024-10-16 13:48:37.260 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.262 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:48:37.262 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:48:37.263 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:48:37.263 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:48:37.267436034+08:00 2024-10-16 13:48:37.264 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:48:37.263+08:00 2024-10-16 13:48:37.266 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:48:37.266 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:48:37.263+08:00 and status AVAILABLE 2024-10-16 13:48:37.267 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:37.269 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:48:37.269 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:37.269 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:48:37.275 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:48:37.278629050+08:00 2024-10-16 13:48:37.275 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:48:37.276 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:48:37.275+08:00 2024-10-16 13:48:37.276 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.277 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:48:37.277 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:37.278 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:48:37.275+08:00 and status AVAILABLE 2024-10-16 13:48:37.280 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:48:37.281 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:48:37.282 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:48:37.284 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:48:37.286 INFO 4486 --- [ocp-async-4,0cf76f8cf6673d8e,c247bd5ab54b7c88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:37.287 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.288 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:37.288 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:37.290 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:37.297 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:37.298 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.299 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:48:37.301 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.303 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:48:37.305 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:37.310 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:37.313 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:48:37.318 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.319 INFO 4486 --- [ocp-schedules-10,1b11c1521f8fece0,c30e51a3195062b3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:48:37.338 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:48:37.346 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:48:37.347 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:48:37.350 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:48:37.356 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:48:37.358 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:48:37.359 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:48:37.368 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:48:37.370 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:37.375 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:37.381 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:37.385 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:48:37.401 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:48:37.408 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:48:37.411 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:48:37.422 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:48:37.423 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:48:37.429 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:37.439 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:48:37.444 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:48:37.446 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:37.456 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:37.457 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:48:37.458 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:48:37.468 INFO 4486 --- [host-check-host-status3,8d1bc6b4364ff304,b74a4071a27aa2ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:48:37.476 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:48:37.481 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:48:37.491 INFO 4486 --- [host-check-host-status4,8d1bc6b4364ff304,49821d2ffc6b172c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:48:37.496 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:48:37.500 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:37.502 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.503 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:48:37.504 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.505 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.506 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:48:37.509 INFO 4486 --- [host-check-host-status2,8d1bc6b4364ff304,76fd45741ffb2cae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:48:37.515 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.515 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.515 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.515 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.515 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.515 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.539 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:48:37.544 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:48:37.552 INFO 4486 --- [host-check-host-status1,8d1bc6b4364ff304,a670e6656dd58242] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:48:37.580 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:48:37.629 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:37.649 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:37.650 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.651 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:48:37.653 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:48:37.654 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.654 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:48:37.655 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.656 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.657 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:48:37.666 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.666 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.666 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.666 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.666 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.666 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.675 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:48:37.767 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,fbeba5b654f47072,2db130ab650291ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fbeba5b654f47072, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:37.767 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,fbeba5b654f47072,2db130ab650291ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fbeba5b654f47072, duration=2 ms] 2024-10-16 13:48:37.869 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:48:37.914 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:37.916 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.917 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:48:37.918 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.919 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.920 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:48:37.928 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.928 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.928 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.928 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.928 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:37.928 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:37.949 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:48:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:48:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:48:37.994 INFO 4486 --- [prometheus-minute-schedule-1,869e696f8c795f58,2a22ba3254c04492] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:38.001 INFO 4486 --- [prometheus-minute-schedule-1,869e696f8c795f58,2a22ba3254c04492] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:38.109 INFO 4486 --- [host-check-clockdiff4,79258b49414bfe15,de60a1cb047d47a4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057718 4 4 , error= 2024-10-16 13:48:38.171 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:48:38.213 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:38.215 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:38.216 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:48:38.217 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:38.217 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:38.219 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:48:38.228 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:38.228 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:38.228 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:38.228 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:38.228 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:48:38.230 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:38.293 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:48:38.371 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bf51e90fb82c93c1,bf0fa42b43814ce4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=bf51e90fb82c93c1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:48:38.371 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bf51e90fb82c93c1,bf0fa42b43814ce4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=bf51e90fb82c93c1, duration=1 ms] 2024-10-16 13:48:38.502 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:48:38.298709, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:48:38.502 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:48:38.298709, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:48:38.539 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:48:38.298709} } 2024-10-16 13:48:38.598 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:48:38.605 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:48:38.607 INFO 4486 --- [ocp-schedules-5,67323c7e502f691b,a1d289dab1bae55b] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:48:38.852 INFO 4486 --- [ocp-schedules-22,99ae1517363a1589,7fc4cd9c6c1eff7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:48:39.072 INFO 4486 --- [ocp-schedules-14,03d1a370c5dfafe6,d410988829c4775c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:48:39.286 INFO 4486 --- [ocp-async-10,0283f8507d67a6cc,baca38158fb30122] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:39.947 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c646b7e0afc17c59,1e0f787055010af5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c646b7e0afc17c59, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:39.948 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c646b7e0afc17c59,1e0f787055010af5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c646b7e0afc17c59, duration=1 ms] 2024-10-16 13:48:40.007 INFO 4486 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:48:40.007 INFO 4486 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:48:40.027 INFO 4486 --- [Check obproxy1,baf4c99521178fac,770f92a8f9c83e09] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:48:40.027 INFO 4486 --- [Check obproxy1,baf4c99521178fac,770f92a8f9c83e09] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:40.028 INFO 4486 --- [Check obproxy2,eb4fa40f3ba05f20,c9413dd12c3465a8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:48:40.028 INFO 4486 --- [Check obproxy2,eb4fa40f3ba05f20,c9413dd12c3465a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:40.033 INFO 4486 --- [Collect obproxy1,46228624ff09f2b4,aa055d103694bc83] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:48:40.034 INFO 4486 --- [Collect obproxy1,46228624ff09f2b4,aa055d103694bc83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:40.034 INFO 4486 --- [Collect obproxy2,5d68ebd30c19ebbf,ee760f29bedd377b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:48:40.035 INFO 4486 --- [Collect obproxy2,5d68ebd30c19ebbf,ee760f29bedd377b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:48:40.052 INFO 4486 --- [Check obproxy1,baf4c99521178fac,770f92a8f9c83e09] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:48:40.052 INFO 4486 --- [Check obproxy1,baf4c99521178fac,770f92a8f9c83e09] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:48:40.053 INFO 4486 --- [Check obproxy2,eb4fa40f3ba05f20,c9413dd12c3465a8] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:48:40.053 INFO 4486 --- [Check obproxy2,eb4fa40f3ba05f20,c9413dd12c3465a8] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:48:40.054 INFO 4486 --- [Collect obproxy1,46228624ff09f2b4,aa055d103694bc83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:48:40.057 INFO 4486 --- [Collect obproxy1,46228624ff09f2b4,aa055d103694bc83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:48:40.058 INFO 4486 --- [Collect obproxy2,5d68ebd30c19ebbf,ee760f29bedd377b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:48:40.060 INFO 4486 --- [Collect obproxy1,46228624ff09f2b4,aa055d103694bc83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:48:40.063 INFO 4486 --- [Collect obproxy2,5d68ebd30c19ebbf,ee760f29bedd377b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:48:40.067 INFO 4486 --- [Collect obproxy2,5d68ebd30c19ebbf,ee760f29bedd377b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:48:40.228 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:40.833 INFO 4486 --- [ocp-schedules-26,e1b77f9b9c9a195e,8af746b261fabcca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:48:40.994 INFO 4486 --- [prometheus-minute-schedule-1,d9ab74771d9b3c3c,da27fdaed89d3c69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:40.995 INFO 4486 --- [prometheus-hour-schedule-1,a0e40b359bbd34dd,3cf7c9c2cf7d37f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:48:41.000 INFO 4486 --- [prometheus-minute-schedule-1,d9ab74771d9b3c3c,da27fdaed89d3c69] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:41.286 INFO 4486 --- [ocp-async-5,278bd72d3ebf8992,75ddb1aa135b3bc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:41.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f1bcf1be17e365a7,0d15f122aa30e5b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f1bcf1be17e365a7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:41.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f1bcf1be17e365a7,0d15f122aa30e5b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f1bcf1be17e365a7, duration=0 ms] 2024-10-16 13:48:41.799 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e8254c84736a3e16,fe1ff089b1198c0f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e8254c84736a3e16, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:41.799 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e8254c84736a3e16,fe1ff089b1198c0f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e8254c84736a3e16, duration=1 ms] 2024-10-16 13:48:42.134 INFO 4486 --- [ocp-schedules-9,41331abf38b1e8ba,6fcea4a43ffaf7ed] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:48:43.009 INFO 4486 --- [ocp-async-2,ddbc3666783993d5,cde926fdab6d5914] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:48:43.229 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:43.287 INFO 4486 --- [ocp-async-8,f5690a3c19ea6342,314b40f07182c074] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:44.037 INFO 4486 --- [prometheus-minute-schedule-1,218764e22e57156b,82b7beb07cef2459] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:44.044 INFO 4486 --- [prometheus-minute-schedule-1,218764e22e57156b,82b7beb07cef2459] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:45.287 INFO 4486 --- [ocp-async-6,ca6cf6c2f283903e,355e142c47c9ae55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:45.525 INFO 4486 --- [ocp-schedules-4,9fb12f0a2f9cc81c,710f88148a383496] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:48:45.530 INFO 4486 --- [ocp-schedules-4,9fb12f0a2f9cc81c,710f88148a383496] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:48:45.530 INFO 4486 --- [ocp-schedules-4,9fb12f0a2f9cc81c,710f88148a383496] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:48:45.547 INFO 4486 --- [ocp-schedules-6,aa1dce2e442fcfd1,b5fb66656527fb84] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:48:45.547 INFO 4486 --- [ocp-schedules-6,aa1dce2e442fcfd1,b5fb66656527fb84] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:48:46.231 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:46.994 INFO 4486 --- [prometheus-minute-schedule-1,77d07958c016478f,d8883301fae426a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:47.002 INFO 4486 --- [prometheus-minute-schedule-1,77d07958c016478f,d8883301fae426a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:47.084 INFO 4486 --- [ocp-schedules-2,f3c5efd0463011ba,793875d25cb250c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:48:47.188 INFO 4486 --- [ocp-schedules-27,90f71961a509c0a8,e8a6af2b379b6b58] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:48:47.248 INFO 4486 --- [ocp-schedules-23,755f949a827cc4b4,7224e9cd77579ec0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:48:47.295 INFO 4486 --- [ocp-schedules-15,bd579e56d0cd4e50,d01616fac3909620] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:48:47.295 INFO 4486 --- [ocp-async-2,85b7d5d24890842c,4460f39ffa804d61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:47.299 WARN 4486 --- [scanner-13288247-1,755f949a827cc4b4,1827e9c28ee21c24] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:48:47.375 INFO 4486 --- [alarm-detect-17,755f949a827cc4b4,3908d4371c8b6915] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057695 ~ 1729057717 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057710, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057710, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057710=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:48:47.774 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eaabf2ad91b0355d,d28486d1a5dbae54] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eaabf2ad91b0355d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:47.775 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eaabf2ad91b0355d,d28486d1a5dbae54] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eaabf2ad91b0355d, duration=3 ms] 2024-10-16 13:48:49.232 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:49.295 INFO 4486 --- [ocp-async-4,764133a9bd6df6fc,10b9cc51d9966b82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:49.952 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bbb83b6eecf25b5,7d92667798edafd8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6bbb83b6eecf25b5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:49.952 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bbb83b6eecf25b5,7d92667798edafd8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6bbb83b6eecf25b5, duration=1 ms] 2024-10-16 13:48:49.994 INFO 4486 --- [prometheus-minute-schedule-1,6ebb85af9ef235c0,04b334fb900a9a83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:50.001 INFO 4486 --- [prometheus-minute-schedule-1,6ebb85af9ef235c0,04b334fb900a9a83] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:50.481 INFO 4486 --- [ocp-schedules-4,a32d7296af9499c1,f1b64607bd080291] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:48:50.481 INFO 4486 --- [ocp-schedules-4,a32d7296af9499c1,f1b64607bd080291] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:48:50.483 INFO 4486 --- [ocp-schedules-4,a32d7296af9499c1,f1b64607bd080291] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:48:50.483 INFO 4486 --- [ocp-schedules-4,a32d7296af9499c1,f1b64607bd080291] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:48:50.483 INFO 4486 --- [ocp-schedules-4,a32d7296af9499c1,f1b64607bd080291] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:48:50.483 INFO 4486 --- [ocp-schedules-4,a32d7296af9499c1,f1b64607bd080291] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:48:50.716 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:48:50.716 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:48:50.731 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:48:50.731 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:48:50.731 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:48:50.742 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:48:50.742 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:48:50.742 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:48:50.747 INFO 4486 --- [ocp-schedules-24,7124aa8667686795,391a03bca5e26e6b] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:48:50.809 INFO 4486 --- [ocp-schedules-23,d19d592baa7a2574,7fe136c9859eca2f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:48:50.809 INFO 4486 --- [ocp-schedules-23,d19d592baa7a2574,7fe136c9859eca2f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:48:50.809 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:48:50.809 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:48:50.809 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:48:50.809 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:48:50.809 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:48:50.815 INFO 4486 --- [ocp-schedules-23,d19d592baa7a2574,7fe136c9859eca2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:48:50.815 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:48:50.815 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:48:50.816 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:50.816 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:48:50.816 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:48:50.817 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:48:50.817 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:48:50.817 INFO 4486 --- [ocp-schedules-23,d19d592baa7a2574,7fe136c9859eca2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:48:50.817 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:50.818 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:48:50.818 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:48:50.840 INFO 4486 --- [ocp-schedules-13,6a8947f4c952d741,072b8c30ffbef87f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:48:50.842 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:48:50.842 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:48:50.843 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:48:50.844 INFO 4486 --- [ocp-schedules-23,d19d592baa7a2574,7fe136c9859eca2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:48:50.844 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:48:50.845 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:48:50.851 INFO 4486 --- [ocp-schedules-23,d19d592baa7a2574,7fe136c9859eca2f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:48:51.295 INFO 4486 --- [ocp-async-3,11db8bd31d583637,805f1f560e3e8d55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:51.489 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,05198e8a38ded85c,f14a1f1f80ab91ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=05198e8a38ded85c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:51.489 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,05198e8a38ded85c,f14a1f1f80ab91ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=05198e8a38ded85c, duration=1 ms] 2024-10-16 13:48:51.727 INFO 4486 --- [ocp-schedules-16,76637d0aca4b6d43,9d3344338c1a2038] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:48:51.727 INFO 4486 --- [ocp-schedules-16,ffb66589d9a29697,b1962caef3612c13] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:48:51.729 INFO 4486 --- [ocp-schedules-16,ffb66589d9a29697,b1962caef3612c13] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:48:51.762 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:51.763 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:51.768 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:51.770 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:51.795 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:51.796 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:51.804 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5362f46a2e28b417,15ac2ff0668795a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5362f46a2e28b417, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:51.804 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5362f46a2e28b417,15ac2ff0668795a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5362f46a2e28b417, duration=1 ms] 2024-10-16 13:48:51.805 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:51.806 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:51.806 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:48:51.951 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:51.952 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:51.952 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:51.954 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:48:51.954 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:51.957 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:48:51.961 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:51.971 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:51.971 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:51.981 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:51.989 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:51.990 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:51.991 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:51.991 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:48:51.995 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:48:52.000 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:48:52.001 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:48:52.019 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:52.021 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:52.022 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:48:52.026 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:48:52.060 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:48:52.060 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057140311000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:48:52.083 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:52.084 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:52.085 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:48:52.089 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:52.092 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:48:52.096 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:52.102 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:48:52.112 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:52.113 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:52.113 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:48:52.123 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:52.123 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:52.123 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:48:52.123 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:48:52.151 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:52.152 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:52.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:52.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:52.204 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:52.204 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:48:52.229 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:52.229 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:48:52.233 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:52.257 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:48:52.285 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:52.287 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:52.288 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:48:52.293 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:48:52.297 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:48:52.302 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:48:52.304 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:48:52.327 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:48:52.479 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:48:52.508 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:48:52.510 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:48:52.511 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:48:52.543 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:48:52.543 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:48:52.994 INFO 4486 --- [prometheus-minute-schedule-1,955e78ab1513da6b,8661811e303e4c08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:53.000 INFO 4486 --- [prometheus-minute-schedule-1,955e78ab1513da6b,8661811e303e4c08] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:53.009 INFO 4486 --- [ocp-async-10,0e5eae24542007cc,65b5b2d59a31d3da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:48:53.296 INFO 4486 --- [ocp-async-5,635f4cf6dee016f6,47c7bca68bbbcf3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:54.614 INFO 4486 --- [ocp-schedules-6,5086eed5df30a857,e9d8e4165e2d4841] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:48:54.614+08:00, tenantSize=5 2024-10-16 13:48:54.620 INFO 4486 --- [ocp-schedules-6,5086eed5df30a857,e9d8e4165e2d4841] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729057734614000 2024-10-16 13:48:54.620 INFO 4486 --- [ocp-schedules-6,5086eed5df30a857,e9d8e4165e2d4841] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729057434584000 2024-10-16 13:48:54.638 INFO 4486 --- [ocp-schedules-6,5086eed5df30a857,e9d8e4165e2d4841] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:48:54.644 INFO 4486 --- [ocp-schedules-6,5086eed5df30a857,e9d8e4165e2d4841] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729057734641000 2024-10-16 13:48:54.644 INFO 4486 --- [ocp-schedules-6,5086eed5df30a857,e9d8e4165e2d4841] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729057434543000 2024-10-16 13:48:54.675 INFO 4486 --- [ocp-schedules-6,5086eed5df30a857,e9d8e4165e2d4841] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:48:55.234 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:55.296 INFO 4486 --- [ocp-async-8,fc6b1c4627b39142,12828ad65d403fd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:55.994 INFO 4486 --- [prometheus-minute-schedule-1,724dd825df6ebcb8,cf4c62d0ed2d2bdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:55.999 INFO 4486 --- [prometheus-minute-schedule-1,724dd825df6ebcb8,cf4c62d0ed2d2bdc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:57.296 INFO 4486 --- [ocp-async-10,2ac35ee213a6912d,42272007115ec30f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:57.456 INFO 4486 --- [ocp-schedules-12,5eb128a34b7b1ad8,3a073f72f431b645] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:48:57.482 WARN 4486 --- [scanner-13288574-1,5eb128a34b7b1ad8,b6ada4e0b224f985] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:48:57.535 INFO 4486 --- [alarm-detect-2,5eb128a34b7b1ad8,a14afcb4eab9e475] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057710 ~ 1729057727 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057725, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057725, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057725=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:48:57.782 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c3a7223428ff7dd2,0b14b2ba3aa4c47b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c3a7223428ff7dd2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:57.783 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c3a7223428ff7dd2,0b14b2ba3aa4c47b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c3a7223428ff7dd2, duration=2 ms] 2024-10-16 13:48:58.235 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:48:58.994 INFO 4486 --- [prometheus-minute-schedule-1,7cc59ebfcf95d14e,f11e54f84cd8ea3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057740, interval=60 2024-10-16 13:48:58.999 INFO 4486 --- [prometheus-minute-schedule-1,7cc59ebfcf95d14e,f11e54f84cd8ea3c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:48:59.077 INFO 4486 --- [ocp-schedules-22,2a0a4e845fae3256,42ef59834a779386] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:48:59.296 INFO 4486 --- [ocp-async-7,71b1e175cf7288d1,9377dda6796d8e2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:48:59.958 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,8bd491a5545e3bea,e32c83edc9f0aaa6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8bd491a5545e3bea, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:48:59.958 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,8bd491a5545e3bea,e32c83edc9f0aaa6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8bd491a5545e3bea, duration=2 ms] 2024-10-16 13:49:00.307 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,220411dbe19b7655,eaf4d9d96477b7c2] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obce_3zones&version=2&ObClusterId=1701313639, client=10.197.49.176, traceId=220411dbe19b7655, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obce_3zones,1701313639,2,{"ObRegion":"obce_3zones","ObCluster":"obce_3zones","ObRegionId":1701313639,"ObClusterId":1701313639,"Type":"PRIMARY","timestamp":1729057740307325,"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:49:00.312 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,220411dbe19b7655,eaf4d9d96477b7c2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=obce_3zones, obClusterId=1701313639, rootServerJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, groupName=obce_3zones, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:49:00.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,220411dbe19b7655,eaf4d9d96477b7c2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:49:00.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,220411dbe19b7655,eaf4d9d96477b7c2] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=obce_3zones, clusterId=1701313639, affectRows=1 2024-10-16 13:49:00.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,220411dbe19b7655,eaf4d9d96477b7c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.176, traceId=220411dbe19b7655, duration=8 ms] 2024-10-16 13:49:00.846 INFO 4486 --- [ocp-schedules-14,7233216962d20bb8,0a27ff73a05074d4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:49:01.236 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:01.297 INFO 4486 --- [ocp-async-9,b9aba6a1ca3bf52c,26d02f931c032fb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:01.494 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,38d8ef0aaebd9bb1,ccc04e2d92c8a6cb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=38d8ef0aaebd9bb1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:01.494 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,38d8ef0aaebd9bb1,ccc04e2d92c8a6cb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=38d8ef0aaebd9bb1, duration=2 ms] 2024-10-16 13:49:01.808 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9cbdd19e8858d426,dc04df4bc67e53d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9cbdd19e8858d426, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:01.809 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9cbdd19e8858d426,dc04df4bc67e53d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9cbdd19e8858d426, duration=1 ms] 2024-10-16 13:49:01.978 INFO 4486 --- [ocp-schedules-23,1a0e7dc53a04ec76,96af1e1e068db02a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:49:01.978+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:49:01.982 INFO 4486 --- [ocp-schedules-23,1a0e7dc53a04ec76,96af1e1e068db02a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:49:01.992 INFO 4486 --- [ocp-schedules-23,1a0e7dc53a04ec76,96af1e1e068db02a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:49:01.994 INFO 4486 --- [prometheus-minute-schedule-1,6d2a38a665d0d1b8,70bfa20fb1a3cf29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:02.003 INFO 4486 --- [prometheus-minute-schedule-1,6d2a38a665d0d1b8,70bfa20fb1a3cf29] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:02.008 INFO 4486 --- [ocp-schedules-23,1a0e7dc53a04ec76,96af1e1e068db02a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:49:03.010 INFO 4486 --- [ocp-async-1,c3aa3e6f89042320,ccd27862629a011f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:49:03.297 INFO 4486 --- [ocp-async-10,9863f540900146ac,03379dc0830a49cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:04.238 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:04.674 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1573b85eceebcfa4,96874d57db1e350e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=1573b85eceebcfa4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:49:04.674 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1573b85eceebcfa4,96874d57db1e350e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=1573b85eceebcfa4, duration=1 ms] 2024-10-16 13:49:04.994 INFO 4486 --- [prometheus-minute-schedule-1,672a90e4926ceeff,44c0e8eb4cea1210] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:05.000 INFO 4486 --- [prometheus-minute-schedule-1,672a90e4926ceeff,44c0e8eb4cea1210] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:05.297 INFO 4486 --- [ocp-async-7,1727f83dbe38ed2e,cbabf93ee1d009e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:07.066 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:49:07.067 INFO 4486 --- [ocp-schedules-6,64beb9d094e15f1b,eac9a0f2342c4f47] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:49:07.069 INFO 4486 --- [ocp-schedules-6,64beb9d094e15f1b,eac9a0f2342c4f47] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:49:07.081 INFO 4486 --- [ocp-schedules-3,1f85b7461afa8970,d93b8734908de5a6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:49:07.084 INFO 4486 --- [ocp-schedules-3,1f85b7461afa8970,d93b8734908de5a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:49:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:49:07.091 INFO 4486 --- [ocp-schedules-14,b4f9487348618464,ced768b9d22039e5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:49:07.091 INFO 4486 --- [ocp-schedules-12,4940b039e8727bce,0973e82791b4ef66] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:49:07.091 INFO 4486 --- [ocp-schedules-26,5f15f1b8243f05aa,159fa524fdda9854] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:49:07.092 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:49:07.092 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:49:07.092 INFO 4486 --- [host-check-clockdiff2,b4f9487348618464,8692fc6e0fa9ab62] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:49:07.092 INFO 4486 --- [host-check-clockdiff1,b4f9487348618464,3554e665bfc8ca18] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:49:07.094 INFO 4486 --- [host-check-agent-status1,5f15f1b8243f05aa,9f6e00e69947b552] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:49:07.095 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:49:07.095 INFO 4486 --- [host-check-clockdiff3,b4f9487348618464,9bae4d258005e9d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:49:07.095 INFO 4486 --- [host-check-clockdiff4,b4f9487348618464,f44db08d8e1d95c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:49:07.096 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:49:07.096 INFO 4486 --- [host-check-agent-status2,5f15f1b8243f05aa,a120b08f8ffa70ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:49:07.097 INFO 4486 --- [host-check-agent-status4,5f15f1b8243f05aa,89abdf2f7ebd84cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:49:07.097 INFO 4486 --- [host-check-agent-status3,5f15f1b8243f05aa,aa1b4a8fba996fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:49:07.103 INFO 4486 --- [host-check-agent-status1,5f15f1b8243f05aa,9f6e00e69947b552] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:07.103 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:49:07.104 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:49:07.113 INFO 4486 --- [host-check-agent-status3,5f15f1b8243f05aa,aa1b4a8fba996fbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:49:07.113 INFO 4486 --- [host-check-agent-status1,5f15f1b8243f05aa,9f6e00e69947b552] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:07.113 INFO 4486 --- [host-check-agent-status2,5f15f1b8243f05aa,a120b08f8ffa70ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:49:07.113 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:07.114 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:49:07.114 INFO 4486 --- [host-check-agent-status4,5f15f1b8243f05aa,89abdf2f7ebd84cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:49:07.129 INFO 4486 --- [host-check-agent-status3,5f15f1b8243f05aa,aa1b4a8fba996fbd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:49:07.129 INFO 4486 --- [host-check-clockdiff1,b4f9487348618464,3554e665bfc8ca18] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057747 0 0 , error= 2024-10-16 13:49:07.129 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:49:07.129 INFO 4486 --- [host-check-agent-status2,5f15f1b8243f05aa,a120b08f8ffa70ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:49:07.129 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:07.129 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:49:07.139 INFO 4486 --- [host-check-agent-status4,5f15f1b8243f05aa,89abdf2f7ebd84cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:49:07.147 INFO 4486 --- [host-check-clockdiff4,b4f9487348618464,f44db08d8e1d95c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057747 4 4 , error= 2024-10-16 13:49:07.148 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.149 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:07.149 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:49:07.150 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:49:07.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:07.173 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:49:07.173 INFO 4486 --- [host-check-agent-status1,5f15f1b8243f05aa,9f6e00e69947b552] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:49:07.173 INFO 4486 --- [host-check-agent-status3,5f15f1b8243f05aa,aa1b4a8fba996fbd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:49:07.183 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:49:07.185 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:49:07.185 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:07.186 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:49:07.189179258+08:00 2024-10-16 13:49:07.186 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:49:07.186+08:00 2024-10-16 13:49:07.187 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.187 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:49:07.188 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:49:07.188 INFO 4486 --- [host-check-agent-status3,5f15f1b8243f05aa,aa1b4a8fba996fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:49:07.188 INFO 4486 --- [host-check-agent-status3,5f15f1b8243f05aa,aa1b4a8fba996fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:49:07.188 INFO 4486 --- [host-check-agent-status3,5f15f1b8243f05aa,aa1b4a8fba996fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:49:07.188 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:49:07.191725011+08:00 2024-10-16 13:49:07.188 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:49:07.188+08:00 2024-10-16 13:49:07.189 INFO 4486 --- [host-check-agent-status1,5f15f1b8243f05aa,9f6e00e69947b552] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:49:07.189 INFO 4486 --- [host-check-agent-status1,5f15f1b8243f05aa,9f6e00e69947b552] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:49:07.189 INFO 4486 --- [host-check-agent-status1,5f15f1b8243f05aa,9f6e00e69947b552] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:49:07.190 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:49:07.189728759+08:00 2024-10-16 13:49:07.190 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:49:07.186+08:00 and status AVAILABLE 2024-10-16 13:49:07.190 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:49:07.190+08:00 2024-10-16 13:49:07.190 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:49:07.193632990+08:00 2024-10-16 13:49:07.191 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:49:07.190+08:00 2024-10-16 13:49:07.191 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:49:07.188+08:00 and status AVAILABLE 2024-10-16 13:49:07.191 INFO 4486 --- [host-check-agent-status2,5f15f1b8243f05aa,a120b08f8ffa70ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:49:07.196 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:49:07.192 INFO 4486 --- [host-check-agent-status4,5f15f1b8243f05aa,89abdf2f7ebd84cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:49:07.196 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:49:07.190+08:00 and status AVAILABLE 2024-10-16 13:49:07.196 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:49:07.190+08:00 and status AVAILABLE 2024-10-16 13:49:07.197 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:49:07.198 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:49:07.198 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:49:07.199 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:49:07.200 INFO 4486 --- [host-check-agent-status2,5f15f1b8243f05aa,a120b08f8ffa70ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:49:07.200 INFO 4486 --- [host-check-agent-status2,5f15f1b8243f05aa,a120b08f8ffa70ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:49:07.200 INFO 4486 --- [host-check-agent-status2,5f15f1b8243f05aa,a120b08f8ffa70ab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:49:07.202 INFO 4486 --- [host-check-agent-status4,5f15f1b8243f05aa,89abdf2f7ebd84cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:49:07.202 INFO 4486 --- [host-check-agent-status4,5f15f1b8243f05aa,89abdf2f7ebd84cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:49:07.202 INFO 4486 --- [host-check-agent-status4,5f15f1b8243f05aa,89abdf2f7ebd84cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:49:07.203 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.204 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:49:07.207 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:49:07.207 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:07.207 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:49:07.208 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:49:07.208 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:49:07.209 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:49:07.209 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:07.209 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:49:07.214 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.215 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:49:07.227 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.229 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:49:07.232 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:07.232 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.232 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:07.233 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:49:07.234 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:07.235 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:07.239 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:07.260 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:07.262 INFO 4486 --- [ocp-schedules-28,e238cd659fd0aa31,0862e9eda2b3d695] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:49:07.266 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:49:07.268 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:49:07.272 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:49:07.277 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:49:07.278 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:49:07.282 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:49:07.291 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:07.292 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:07.296 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:07.297 INFO 4486 --- [ocp-async-9,6f3b0c8aa6929e22,b6ccf40d6f7e1cac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:07.299 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:49:07.310 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:49:07.320 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:49:07.325 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:07.326 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:49:07.329 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:49:07.330 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:49:07.337 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:49:07.338 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:49:07.344 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:07.353 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:07.353 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:07.374 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:49:07.377 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:49:07.384 INFO 4486 --- [host-check-host-status3,4940b039e8727bce,82026f329945e6c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:49:07.388 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:49:07.389 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:49:07.392 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:49:07.400 INFO 4486 --- [host-check-host-status2,4940b039e8727bce,b7b67de93276735b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:49:07.402 INFO 4486 --- [host-check-host-status4,4940b039e8727bce,4a4c50754ed7913c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:49:07.404 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:07.456 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:49:07.466 INFO 4486 --- [host-check-host-status1,4940b039e8727bce,7598031ad611102d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:49:07.622 INFO 4486 --- [ocp-schedules-27,c5a111492b9b3b97,2321c52bfd6412b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:49:07.659 WARN 4486 --- [scanner-13288813-1,c5a111492b9b3b97,e1bd9b608e3643eb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:49:07.802 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,91136ffddb2931a7,c54079382e5191f2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=91136ffddb2931a7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:07.803 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,91136ffddb2931a7,c54079382e5191f2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=91136ffddb2931a7, duration=13 ms] 2024-10-16 13:49:07.994 INFO 4486 --- [prometheus-minute-schedule-1,8d70e2104f958490,557fd2a796d42e4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:08.000 INFO 4486 --- [prometheus-minute-schedule-1,8d70e2104f958490,557fd2a796d42e4e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:08.053 INFO 4486 --- [host-check-clockdiff3,b4f9487348618464,9bae4d258005e9d0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057748 4 4 , error= 2024-10-16 13:49:09.297 INFO 4486 --- [ocp-async-6,04be5f217689959b,5deb90242641ca0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:09.640 INFO 4486 --- [host-check-clockdiff2,b4f9487348618464,8692fc6e0fa9ab62] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057749 4 4 , error= 2024-10-16 13:49:09.964 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5be1c7a80680fe2f,53f3a60ff1577f3d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5be1c7a80680fe2f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:09.964 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5be1c7a80680fe2f,53f3a60ff1577f3d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5be1c7a80680fe2f, duration=1 ms] 2024-10-16 13:49:10.007 INFO 4486 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:49:10.007 INFO 4486 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:49:10.025 INFO 4486 --- [Check obproxy1,1b968c09c093e140,464e2294c5888f7b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:49:10.025 INFO 4486 --- [Check obproxy1,1b968c09c093e140,464e2294c5888f7b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:10.025 INFO 4486 --- [Check obproxy2,2b82de85b1baea47,9f7899859802c175] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:49:10.025 INFO 4486 --- [Check obproxy2,2b82de85b1baea47,9f7899859802c175] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:10.030 INFO 4486 --- [Collect obproxy1,8d93359d60616bb4,4ec76a2076ed4c10] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:49:10.030 INFO 4486 --- [Collect obproxy1,8d93359d60616bb4,4ec76a2076ed4c10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:10.032 INFO 4486 --- [Collect obproxy2,646d52527b68915c,5e829617a39a1d9d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:49:10.032 INFO 4486 --- [Collect obproxy2,646d52527b68915c,5e829617a39a1d9d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:10.048 INFO 4486 --- [Check obproxy2,2b82de85b1baea47,9f7899859802c175] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:49:10.048 INFO 4486 --- [Check obproxy2,2b82de85b1baea47,9f7899859802c175] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:49:10.049 INFO 4486 --- [Check obproxy1,1b968c09c093e140,464e2294c5888f7b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:49:10.049 INFO 4486 --- [Check obproxy1,1b968c09c093e140,464e2294c5888f7b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:49:10.054 INFO 4486 --- [Collect obproxy2,646d52527b68915c,5e829617a39a1d9d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:49:10.056 INFO 4486 --- [Collect obproxy1,8d93359d60616bb4,4ec76a2076ed4c10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:49:10.059 INFO 4486 --- [Collect obproxy1,8d93359d60616bb4,4ec76a2076ed4c10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:49:10.059 INFO 4486 --- [Collect obproxy2,646d52527b68915c,5e829617a39a1d9d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:49:10.062 INFO 4486 --- [Collect obproxy1,8d93359d60616bb4,4ec76a2076ed4c10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:49:10.064 INFO 4486 --- [Collect obproxy2,646d52527b68915c,5e829617a39a1d9d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:49:10.240 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:10.852 INFO 4486 --- [ocp-schedules-10,0697cb46fb194b74,701dd4cca1e488f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:49:10.978 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:49:10.994 INFO 4486 --- [prometheus-minute-schedule-1,5883a0989a12ef18,7a7b7b67be2b6709] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:10.999 INFO 4486 --- [prometheus-minute-schedule-1,5883a0989a12ef18,7a7b7b67be2b6709] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:11.008 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:49:11.298 INFO 4486 --- [ocp-async-2,aea42331ba089fa6,5ffde87e30afd4c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:11.497 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,21d81f216ad9de27,1b18a627499433e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=21d81f216ad9de27, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:11.497 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,21d81f216ad9de27,1b18a627499433e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=21d81f216ad9de27, duration=0 ms] 2024-10-16 13:49:11.812 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,436d05e38491471c,3859e5abe009e5ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=436d05e38491471c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:11.812 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,436d05e38491471c,3859e5abe009e5ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=436d05e38491471c, duration=1 ms] 2024-10-16 13:49:13.010 INFO 4486 --- [ocp-async-3,62959408bec1bb56,05b5602103a0da55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:49:13.241 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:13.298 INFO 4486 --- [ocp-async-9,6bb4a96e9cce9a4f,9110fc0c8783d0b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:13.994 INFO 4486 --- [prometheus-minute-schedule-1,01dadbe2879de97c,18ae253ab3350187] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:14.001 INFO 4486 --- [prometheus-minute-schedule-1,01dadbe2879de97c,18ae253ab3350187] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:14.083 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057680, nodeCount=450, affectRows=450 2024-10-16 13:49:14.084 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057680, nodeCount=449, validCount=449 2024-10-16 13:49:14.103 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057680, nodeCount=1000, affectRows=1000 2024-10-16 13:49:14.106 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057680, nodeCount=1000, validCount=1000 2024-10-16 13:49:14.107 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057680, nodeCount=1000, affectRows=1000 2024-10-16 13:49:14.110 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057680, nodeCount=999, validCount=999 2024-10-16 13:49:14.113 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057680, nodeCount=1000, affectRows=1000 2024-10-16 13:49:14.117 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057680, nodeCount=999, validCount=999 2024-10-16 13:49:14.941 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057680, nodeCount=57, affectRows=57 2024-10-16 13:49:14.945 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057680, nodeCount=121, affectRows=121 2024-10-16 13:49:14.947 INFO 4486 --- [metric-archive-async-18,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057680, nodeCount=165, affectRows=165 2024-10-16 13:49:14.950 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057680, nodeCount=137, affectRows=137 2024-10-16 13:49:15.045 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057680, nodeCount=57, validCount=57 2024-10-16 13:49:15.142 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057680, nodeCount=121, validCount=121 2024-10-16 13:49:15.162 INFO 4486 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057680, nodeCount=137, validCount=137 2024-10-16 13:49:15.177 INFO 4486 --- [metric-archive-async-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057680, nodeCount=165, validCount=165 2024-10-16 13:49:15.298 INFO 4486 --- [ocp-async-6,0755f33d019edcdf,b4cab86529a752c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:16.242 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:16.994 INFO 4486 --- [prometheus-minute-schedule-1,619bcdea79366d60,a9ea1542018ec51f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:16.999 INFO 4486 --- [prometheus-minute-schedule-1,619bcdea79366d60,a9ea1542018ec51f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:17.071 INFO 4486 --- [ocp-schedules-6,64beb9d094e15f1b,eac9a0f2342c4f47] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:49:17.184 INFO 4486 --- [ocp-schedules-3,1f85b7461afa8970,d93b8734908de5a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:49:17.191 INFO 4486 --- [ocp-schedules-9,a7ccc618982fd0e9,9807f2c2177d73eb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:49:17.298 INFO 4486 --- [ocp-async-3,b3db09e33ccad4ba,969c2218404c1839] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:17.813 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0931d88ddbf4afe4,df54e331e8ebd6ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0931d88ddbf4afe4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:17.813 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0931d88ddbf4afe4,df54e331e8ebd6ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0931d88ddbf4afe4, duration=3 ms] 2024-10-16 13:49:17.850 INFO 4486 --- [ocp-schedules-19,b691414d05fb1ccb,30d00ae6911b90d0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:49:17.911 WARN 4486 --- [scanner-13289979-1,b691414d05fb1ccb,dc1f6678355fef62] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:49:17.955 INFO 4486 --- [alarm-detect-1,b691414d05fb1ccb,ed2094212304f8e6] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057725 ~ 1729057747 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057740, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057740, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057740=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:49:19.081 INFO 4486 --- [ocp-schedules-22,aa28103fb26e51ab,863e7a93d61d5673] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:49:19.243 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:19.298 INFO 4486 --- [ocp-async-1,1433e6ce82ddfd46,652ff24b23f5f0d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:19.767 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,56b881af84d87bab,24e7544befed3410] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=56b881af84d87bab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:49:19.767 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,56b881af84d87bab,24e7544befed3410] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=56b881af84d87bab, duration=1 ms] 2024-10-16 13:49:19.968 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bc6b00b18b514cf9,31b8b13d15fcd2bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bc6b00b18b514cf9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:19.968 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bc6b00b18b514cf9,31b8b13d15fcd2bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bc6b00b18b514cf9, duration=1 ms] 2024-10-16 13:49:19.994 INFO 4486 --- [prometheus-minute-schedule-1,2d1776836709c5ef,f8095c548005d1df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:20.001 INFO 4486 --- [prometheus-minute-schedule-1,2d1776836709c5ef,f8095c548005d1df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:20.858 INFO 4486 --- [ocp-schedules-9,92a751d340e9fd02,2e3629335f9fd859] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:49:20.996 INFO 4486 --- [prometheus-cache-monitor1,35f41260473a68f1,60614d306e689bdf] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:49:20.996 INFO 4486 --- [prometheus-cache-monitor1,35f41260473a68f1,60614d306e689bdf] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:49:21.299 INFO 4486 --- [ocp-async-8,bc8679629bd47a5a,69ea6352b83e32e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:21.502 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,39bb56cc47cf20d7,b1642b9faf610dc1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=39bb56cc47cf20d7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:21.502 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,39bb56cc47cf20d7,b1642b9faf610dc1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=39bb56cc47cf20d7, duration=1 ms] 2024-10-16 13:49:21.816 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e33e6e9e4a7ed254,773e57ef18869267] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e33e6e9e4a7ed254, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:21.816 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e33e6e9e4a7ed254,773e57ef18869267] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e33e6e9e4a7ed254, duration=1 ms] 2024-10-16 13:49:22.244 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:22.544 INFO 4486 --- [ocp-schedules-16,ffb66589d9a29697,b1962caef3612c13] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:49:22.545 INFO 4486 --- [ocp-schedules-16,6d9c1f978215d05c,0eab68e2db15dfd5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:49:22.547 INFO 4486 --- [ocp-schedules-16,6d9c1f978215d05c,0eab68e2db15dfd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:49:22.577 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.578 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:22.589 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.589 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:22.615 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:22.616 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.617 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:49:22.619 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.620 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:22.646 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:22.647 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.647 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:22.648 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:49:22.648 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.651 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:49:22.654 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:22.664 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.664 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:22.675 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.683 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:22.684 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.685 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:22.685 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:49:22.690 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:49:22.694 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:22.696 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:22.717 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:49:22.731 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:22.733 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.734 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:49:22.784 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:49:22.784 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:49:22.787 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:22.788 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.789 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:22.793 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.797 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:49:22.800 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:22.800 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:49:22.810 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:22.810 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:22.811 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:49:22.821 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:49:22.821 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:49:22.826 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.826 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:22.855 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.855 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:22.880 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.880 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:22.910 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.910 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:22.939 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.939 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:22.969 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:22.994 INFO 4486 --- [prometheus-minute-schedule-1,e9ae34eb009ca622,ea0c4fe80e0bbb4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:22.999 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:23.001 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:23.001 INFO 4486 --- [prometheus-minute-schedule-1,e9ae34eb009ca622,ea0c4fe80e0bbb4e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:23.002 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:23.007 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:49:23.010 INFO 4486 --- [ocp-async-4,4087cffff203ee29,c32322919f18ba53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:49:23.012 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:49:23.019 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:49:23.021 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:49:23.049 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:49:23.180 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:49:23.209 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:23.211 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:23.212 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:49:23.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:49:23.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:49:23.299 INFO 4486 --- [ocp-async-2,ddcc734ff8ed0d9b,4aa675509e23701c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:23.493 INFO 4486 --- [ocp-schedules-29,3f96fc607d38100b,a93507da1741a234] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:49:24.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0649c1c382e10dc0,3805165b68eecafd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=0649c1c382e10dc0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:49:24.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0649c1c382e10dc0,3805165b68eecafd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=0649c1c382e10dc0, duration=2 ms] 2024-10-16 13:49:24.798 INFO 4486 --- [ocp-schedules-11,0dacdd3a4f02edc1,4cc6f76b1c5b7373] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:49:25.245 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:25.299 INFO 4486 --- [ocp-async-7,89ddb3c0c4aec76b,56815069bb9b7415] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:25.803 INFO 4486 --- [ocp-schedules-24,a49729bdd92d8f3a,95404778cbbb301d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:49:25.803 INFO 4486 --- [ocp-schedules-24,a49729bdd92d8f3a,95404778cbbb301d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:49:25.834 INFO 4486 --- [Collect all tenants major compactions2,a49729bdd92d8f3a,e9851fdc94779541] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:49:25.834 INFO 4486 --- [Collect all tenants major compactions1,a49729bdd92d8f3a,49156f4fa8f40e2b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:49:25.834 INFO 4486 --- [Collect all tenants major compactions3,a49729bdd92d8f3a,5d6f290751dea110] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:49:25.835 INFO 4486 --- [Collect all tenants major compactions4,a49729bdd92d8f3a,309f90131be8b79b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:49:25.835 INFO 4486 --- [Collect all tenants major compactions5,a49729bdd92d8f3a,7c3f48b612043070] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:49:25.848 INFO 4486 --- [Collect all tenants major compactions2,a49729bdd92d8f3a,e9851fdc94779541] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:25.848 INFO 4486 --- [Collect all tenants major compactions1,a49729bdd92d8f3a,49156f4fa8f40e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:25.849 INFO 4486 --- [Collect all tenants major compactions2,a49729bdd92d8f3a,e9851fdc94779541] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:49:25.850 INFO 4486 --- [Collect all tenants major compactions1,a49729bdd92d8f3a,49156f4fa8f40e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:49:25.853 INFO 4486 --- [Collect all tenants major compactions2,a49729bdd92d8f3a,e9851fdc94779541] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:49:25.853 INFO 4486 --- [Collect all tenants major compactions1,a49729bdd92d8f3a,49156f4fa8f40e2b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:49:25.853 INFO 4486 --- [Collect all tenants major compactions2,a49729bdd92d8f3a,e9851fdc94779541] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:49:25.853 INFO 4486 --- [Collect all tenants major compactions1,a49729bdd92d8f3a,49156f4fa8f40e2b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:49:25.858 INFO 4486 --- [Collect all tenants major compactions3,a49729bdd92d8f3a,5d6f290751dea110] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:25.860 INFO 4486 --- [Collect all tenants major compactions3,a49729bdd92d8f3a,5d6f290751dea110] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:49:25.865 INFO 4486 --- [Collect all tenants major compactions3,a49729bdd92d8f3a,5d6f290751dea110] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:49:25.865 INFO 4486 --- [Collect all tenants major compactions3,a49729bdd92d8f3a,5d6f290751dea110] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:49:25.868 INFO 4486 --- [Collect all tenants major compactions4,a49729bdd92d8f3a,309f90131be8b79b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:25.869 INFO 4486 --- [Collect all tenants major compactions5,a49729bdd92d8f3a,7c3f48b612043070] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:25.870 INFO 4486 --- [Collect all tenants major compactions4,a49729bdd92d8f3a,309f90131be8b79b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:49:25.870 INFO 4486 --- [Collect all tenants major compactions5,a49729bdd92d8f3a,7c3f48b612043070] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:49:25.875 INFO 4486 --- [Collect all tenants major compactions4,a49729bdd92d8f3a,309f90131be8b79b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:49:25.875 INFO 4486 --- [Collect all tenants major compactions4,a49729bdd92d8f3a,309f90131be8b79b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:49:25.875 INFO 4486 --- [Collect all tenants major compactions5,a49729bdd92d8f3a,7c3f48b612043070] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:49:25.875 INFO 4486 --- [Collect all tenants major compactions5,a49729bdd92d8f3a,7c3f48b612043070] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:49:25.876 INFO 4486 --- [ocp-schedules-24,a49729bdd92d8f3a,95404778cbbb301d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:49:25.994 INFO 4486 --- [prometheus-minute-schedule-1,99020d0cf71c810d,5bbf77e111439dd9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:25.999 INFO 4486 --- [prometheus-minute-schedule-1,99020d0cf71c810d,5bbf77e111439dd9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:27.043 INFO 4486 --- [ocp-schedules-18,66f72d009f5c151a,82f97b7978000d92] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:49:27.299 INFO 4486 --- [ocp-async-4,6952bbcb80d62245,ae016ed864a61dd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:27.455 INFO 4486 --- [ocp-schedules-8,88b8f628f7b9fcae,eff6636d90f1b43e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:49:27.455+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:49:27.457 INFO 4486 --- [ocp-schedules-8,88b8f628f7b9fcae,eff6636d90f1b43e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:49:27.458 INFO 4486 --- [ocp-schedules-8,88b8f628f7b9fcae,eff6636d90f1b43e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:49:27.459 INFO 4486 --- [ocp-schedules-8,88b8f628f7b9fcae,eff6636d90f1b43e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:49:27.819 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,cc32870445d8a8b8,6813603386fee3c5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cc32870445d8a8b8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:27.819 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,cc32870445d8a8b8,6813603386fee3c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cc32870445d8a8b8, duration=1 ms] 2024-10-16 13:49:28.005 INFO 4486 --- [ocp-schedules-13,07646631ef5c45cd,f016a1f7d0a977a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:49:28.018 WARN 4486 --- [scanner-13290160-1,07646631ef5c45cd,c6777ed39ba7ba9f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:49:28.109 INFO 4486 --- [alarm-detect-3,07646631ef5c45cd,efdc24c4ca7b06cf] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057740 ~ 1729057758 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057755, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057755, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057755=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:49:28.246 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:28.994 INFO 4486 --- [prometheus-minute-schedule-1,3f3c2d8103be3289,2e90916fadb0346c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:29.001 INFO 4486 --- [prometheus-minute-schedule-1,3f3c2d8103be3289,2e90916fadb0346c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:29.300 INFO 4486 --- [ocp-async-10,35d2c6042fac358d,d59cc2a0fdbcebfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:29.975 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,58a2558a78810aa5,ea1a7a98d7f48247] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=58a2558a78810aa5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:29.975 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,58a2558a78810aa5,ea1a7a98d7f48247] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=58a2558a78810aa5, duration=2 ms] 2024-10-16 13:49:30.103 INFO 4486 --- [prometheus-second-schedule-1,b4e6af2bc5aa039e,3efe2ecbe65a29f9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:49:30.864 INFO 4486 --- [ocp-schedules-2,d0df3c04a5cf6609,59cc14a690394791] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:49:31.247 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:31.300 INFO 4486 --- [ocp-async-2,19c0d91b6f2f1105,099eef2c0e9c4ebb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:31.507 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5c74761a4686f1d9,2e8d8e41af7e93f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5c74761a4686f1d9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:31.507 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5c74761a4686f1d9,2e8d8e41af7e93f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5c74761a4686f1d9, duration=1 ms] 2024-10-16 13:49:31.821 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,95de8935967b6227,3f94a8f1fd1fb371] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=95de8935967b6227, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:31.822 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,95de8935967b6227,3f94a8f1fd1fb371] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=95de8935967b6227, duration=2 ms] 2024-10-16 13:49:31.994 INFO 4486 --- [prometheus-minute-schedule-1,95d19e31bab90dc1,941fa2e87abbbd83] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:32.000 INFO 4486 --- [prometheus-minute-schedule-1,95d19e31bab90dc1,941fa2e87abbbd83] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:33.010 INFO 4486 --- [ocp-async-5,51eacc61b5920a6e,5fb0bac78f79f028] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:49:33.300 INFO 4486 --- [ocp-async-9,0c395f160f374c9d,59d44056a9d62c82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:33.536 INFO 4486 --- [ocp-schedules-15,bfd7d5aee12af060,bc1df114627324fe] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:49:33.538 INFO 4486 --- [ocp-schedules-15,bfd7d5aee12af060,bc1df114627324fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:49:33.538 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:49:33.538 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:49:33.539 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:49:34.248 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:34.994 INFO 4486 --- [prometheus-minute-schedule-1,c27a6f96b18893b8,8b33c3e456d2677f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:34.999 INFO 4486 --- [prometheus-minute-schedule-1,c27a6f96b18893b8,8b33c3e456d2677f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:35.300 INFO 4486 --- [ocp-async-6,c1d2864f1719000b,8eeb16ea061a94e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:37.063 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:49:37.065 INFO 4486 --- [ocp-schedules-27,647824fc7ef9d38a,094f5877a6daff29] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:49:37.066 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:49:37.066 INFO 4486 --- [ocp-schedules-25,d7e4bc46336bdc65,07714ba7a0b854ec] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:49:37.068 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:49:37.069 INFO 4486 --- [ocp-schedules-25,d7e4bc46336bdc65,07714ba7a0b854ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:49:37.071 INFO 4486 --- [ocp-schedules-27,647824fc7ef9d38a,094f5877a6daff29] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:49:37.081 INFO 4486 --- [ocp-schedules-21,4a2cb6904778a684,eab0953c55d51b56] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:49:37.083 INFO 4486 --- [ocp-schedules-21,4a2cb6904778a684,eab0953c55d51b56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:49:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:49:37.090 INFO 4486 --- [ocp-schedules-11,d926a5b68242c496,668cbbbf2e5b5d54] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:49:37.091 INFO 4486 --- [ocp-schedules-13,3e020b940f2219ca,a0fa4bba994ab038] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:49:37.091 INFO 4486 --- [host-check-agent-status1,d926a5b68242c496,13eda7e9e91b1e89] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:49:37.091 INFO 4486 --- [host-check-agent-status2,d926a5b68242c496,d024bd2f67975549] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:49:37.091 INFO 4486 --- [ocp-schedules-22,e7e728bae2e3fee2,4ff5baeb053f38b4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:49:37.091 INFO 4486 --- [host-check-agent-status3,d926a5b68242c496,3737768b8eba3c97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:49:37.091 INFO 4486 --- [host-check-clockdiff1,3e020b940f2219ca,b5632ef7f9800078] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:49:37.092 INFO 4486 --- [host-check-agent-status4,d926a5b68242c496,bd2595d0588988c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:49:37.092 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:49:37.093 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:49:37.093 INFO 4486 --- [host-check-clockdiff2,3e020b940f2219ca,3d0a0cdcdc861d01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:49:37.094 INFO 4486 --- [host-check-clockdiff3,3e020b940f2219ca,943f31ff981d9301] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:49:37.094 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:49:37.094 INFO 4486 --- [host-check-clockdiff4,3e020b940f2219ca,171ecdda656254eb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:49:37.095 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:49:37.099 INFO 4486 --- [host-check-agent-status3,d926a5b68242c496,3737768b8eba3c97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:49:37.100 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:37.101 INFO 4486 --- [host-check-agent-status2,d926a5b68242c496,d024bd2f67975549] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:49:37.101 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:49:37.101 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:49:37.101 INFO 4486 --- [host-check-agent-status4,d926a5b68242c496,bd2595d0588988c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:49:37.101 INFO 4486 --- [host-check-agent-status1,d926a5b68242c496,13eda7e9e91b1e89] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:37.101 INFO 4486 --- [host-check-agent-status3,d926a5b68242c496,3737768b8eba3c97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:49:37.102 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:37.102 INFO 4486 --- [host-check-agent-status2,d926a5b68242c496,d024bd2f67975549] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:49:37.103 INFO 4486 --- [host-check-agent-status4,d926a5b68242c496,bd2595d0588988c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:49:37.103 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:49:37.103 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:49:37.105 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:49:37.106 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:49:37.103 INFO 4486 --- [host-check-agent-status1,d926a5b68242c496,13eda7e9e91b1e89] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:37.137 INFO 4486 --- [host-check-clockdiff2,3e020b940f2219ca,3d0a0cdcdc861d01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057777 4 4 , error= 2024-10-16 13:49:37.137 INFO 4486 --- [host-check-clockdiff3,3e020b940f2219ca,943f31ff981d9301] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057777 4 4 , error= 2024-10-16 13:49:37.147 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.149 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:37.150 INFO 4486 --- [host-check-clockdiff1,3e020b940f2219ca,b5632ef7f9800078] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057777 0 0 , error= 2024-10-16 13:49:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:49:37.165 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.166 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:37.168 INFO 4486 --- [host-check-agent-status2,d926a5b68242c496,d024bd2f67975549] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:49:37.170 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:49:37.170 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:49:37.170 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:49:37.170 INFO 4486 --- [host-check-agent-status3,d926a5b68242c496,3737768b8eba3c97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:49:37.172 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:49:37.172 INFO 4486 --- [host-check-agent-status4,d926a5b68242c496,bd2595d0588988c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:49:37.172 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:49:37.172 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:49:37.176387729+08:00 2024-10-16 13:49:37.172 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:49:37.172231648+08:00 2024-10-16 13:49:37.173 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:49:37.173+08:00 2024-10-16 13:49:37.173 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:49:37.173+08:00 2024-10-16 13:49:37.174 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:49:37.178037261+08:00 2024-10-16 13:49:37.175 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:49:37.174+08:00 2024-10-16 13:49:37.175 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.175 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:49:37.178609429+08:00 2024-10-16 13:49:37.175 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.175 INFO 4486 --- [host-check-agent-status1,d926a5b68242c496,13eda7e9e91b1e89] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:49:37.175 INFO 4486 --- [host-check-agent-status3,d926a5b68242c496,3737768b8eba3c97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:49:37.175 INFO 4486 --- [host-check-agent-status3,d926a5b68242c496,3737768b8eba3c97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:49:37.175 INFO 4486 --- [host-check-agent-status3,d926a5b68242c496,3737768b8eba3c97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:49:37.175 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:49:37.175+08:00 2024-10-16 13:49:37.176 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:49:37.176 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:49:37.173+08:00 and status AVAILABLE 2024-10-16 13:49:37.176 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:49:37.173+08:00 and status AVAILABLE 2024-10-16 13:49:37.176 INFO 4486 --- [host-check-agent-status2,d926a5b68242c496,d024bd2f67975549] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:49:37.176 INFO 4486 --- [host-check-agent-status2,d926a5b68242c496,d024bd2f67975549] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:49:37.176 INFO 4486 --- [host-check-agent-status2,d926a5b68242c496,d024bd2f67975549] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:49:37.177 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:49:37.174+08:00 and status AVAILABLE 2024-10-16 13:49:37.177 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.177 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.178 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:49:37.175+08:00 and status AVAILABLE 2024-10-16 13:49:37.178 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:49:37.178 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:49:37.179 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:49:37.179 INFO 4486 --- [host-check-agent-status1,d926a5b68242c496,13eda7e9e91b1e89] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:49:37.179 INFO 4486 --- [host-check-agent-status1,d926a5b68242c496,13eda7e9e91b1e89] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:49:37.179 INFO 4486 --- [host-check-agent-status1,d926a5b68242c496,13eda7e9e91b1e89] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:49:37.179 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:49:37.180 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:49:37.180 INFO 4486 --- [host-check-agent-status4,d926a5b68242c496,bd2595d0588988c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:49:37.180 INFO 4486 --- [host-check-agent-status4,d926a5b68242c496,bd2595d0588988c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:49:37.180 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:49:37.180 INFO 4486 --- [host-check-agent-status4,d926a5b68242c496,bd2595d0588988c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:49:37.186 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:37.186 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:49:37.187 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:49:37.188 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:37.188 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:49:37.188 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.188 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:49:37.188 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:49:37.189 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:49:37.189 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:49:37.190 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.190 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.190 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.190 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.190 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.191 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.199 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.200 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:49:37.206 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:37.207 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:49:37.208 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:37.212 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.212 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:37.212 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:49:37.213 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:49:37.214 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.216 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:49:37.226 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.227 INFO 4486 --- [ocp-schedules-8,96407688ab608fab,2a430b87ab936e58] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:49:37.242 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:49:37.247 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:49:37.249 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:37.250 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:49:37.252 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:49:37.256 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:49:37.257 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:49:37.261 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:49:37.265 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:49:37.267 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:37.270 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:37.276 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:37.278 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:49:37.295 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:49:37.300 INFO 4486 --- [ocp-async-7,1b5133cf76ceb808,32caf58cfe4d9b41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:37.304 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:49:37.307 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:49:37.313 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:49:37.314 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:49:37.322 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:37.323 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:49:37.329 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:37.333 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:49:37.336 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:37.343 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:49:37.354 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:49:37.357 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:49:37.362 INFO 4486 --- [host-check-host-status3,e7e728bae2e3fee2,1333434aa220420e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:49:37.367 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:49:37.371 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:49:37.375 INFO 4486 --- [host-check-host-status4,e7e728bae2e3fee2,821687a07b5b0481] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:49:37.380 INFO 4486 --- [host-check-host-status2,e7e728bae2e3fee2,54dc5d7dbb2f67d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:49:37.393 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:37.395 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.396 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:49:37.397 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.399 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.400 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:49:37.409 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:49:37.410 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.411 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.411 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.411 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.411 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.411 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.424 INFO 4486 --- [host-check-host-status1,e7e728bae2e3fee2,ca5b9e61e7bddd70] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:49:37.439 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:49:37.481 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:49:37.534 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:37.549 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:37.551 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.552 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:49:37.553 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:49:37.553 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.554 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:49:37.555 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.556 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.557 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:49:37.569 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.569 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.569 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.569 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.569 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.570 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.577 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:49:37.762 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:49:37.808 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:37.811 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.812 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:49:37.813 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.814 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.815 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:49:37.825 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,255662bf12a45534,dd6b9b115050f68e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=255662bf12a45534, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:37.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,255662bf12a45534,dd6b9b115050f68e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=255662bf12a45534, duration=3 ms] 2024-10-16 13:49:37.826 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.826 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.826 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.826 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.826 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:37.827 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:37.843 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:49:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:49:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:49:37.994 INFO 4486 --- [prometheus-minute-schedule-1,d2a7d849d14d5dc3,5053215eb2044146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:37.999 INFO 4486 --- [prometheus-minute-schedule-1,d2a7d849d14d5dc3,5053215eb2044146] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:38.033 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:49:38.073 INFO 4486 --- [host-check-clockdiff4,3e020b940f2219ca,171ecdda656254eb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057778 4 4 , error= 2024-10-16 13:49:38.110 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:38.113 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:38.114 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:49:38.118 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:38.119 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:38.123 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:49:38.134 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:38.134 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:38.134 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:38.134 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:38.134 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:49:38.134 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:38.201 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:49:38.215 INFO 4486 --- [ocp-schedules-7,0bad9ab67efd5ad5,6d090536c772305f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:49:38.260 WARN 4486 --- [scanner-13290530-1,0bad9ab67efd5ad5,596696e15219eb79] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:49:38.474 INFO 4486 --- [ocp-schedules-20,abc17b1fe03abdb2,3bc147b6ad6b1cae] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:49:38.474 INFO 4486 --- [ocp-schedules-20,abc17b1fe03abdb2,3bc147b6ad6b1cae] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:49:38.476 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:49:38.205006, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:49:38.476 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:49:38.205006, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:49:38.522 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:49:38.205006} } 2024-10-16 13:49:38.591 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:49:38.597 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:49:38.600 INFO 4486 --- [ocp-schedules-17,1f550cdb4a69da76,8c3132d2f61a9fdc] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:49:39.086 INFO 4486 --- [ocp-schedules-23,1f26d79a1ad22ee2,131d913032e792b3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:49:39.301 INFO 4486 --- [ocp-async-9,7a9e2886f63bc246,6431d3c20f3fdc23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:39.981 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,23bbaa25e8fef3fd,cadf08432045702e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=23bbaa25e8fef3fd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:39.981 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,23bbaa25e8fef3fd,cadf08432045702e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=23bbaa25e8fef3fd, duration=2 ms] 2024-10-16 13:49:40.007 INFO 4486 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:49:40.008 INFO 4486 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:49:40.028 INFO 4486 --- [Check obproxy2,4900269360d4b06d,4c75238b4f8fd381] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:49:40.028 INFO 4486 --- [Check obproxy2,4900269360d4b06d,4c75238b4f8fd381] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:40.030 INFO 4486 --- [Check obproxy1,57482e48c724b0ce,4c257ceebd60a34a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:49:40.030 INFO 4486 --- [Check obproxy1,57482e48c724b0ce,4c257ceebd60a34a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:40.034 INFO 4486 --- [Collect obproxy1,70b191e5260217f1,527ca4537d026358] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:49:40.035 INFO 4486 --- [Collect obproxy1,70b191e5260217f1,527ca4537d026358] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:40.035 INFO 4486 --- [Collect obproxy2,e9c273efead4b5cf,279512656ed34104] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:49:40.035 INFO 4486 --- [Collect obproxy2,e9c273efead4b5cf,279512656ed34104] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:49:40.054 INFO 4486 --- [Check obproxy2,4900269360d4b06d,4c75238b4f8fd381] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:49:40.054 INFO 4486 --- [Check obproxy2,4900269360d4b06d,4c75238b4f8fd381] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:49:40.055 INFO 4486 --- [Collect obproxy2,e9c273efead4b5cf,279512656ed34104] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:49:40.061 INFO 4486 --- [Collect obproxy2,e9c273efead4b5cf,279512656ed34104] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:49:40.066 INFO 4486 --- [Collect obproxy2,e9c273efead4b5cf,279512656ed34104] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:49:40.067 INFO 4486 --- [Check obproxy1,57482e48c724b0ce,4c257ceebd60a34a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:49:40.067 INFO 4486 --- [Check obproxy1,57482e48c724b0ce,4c257ceebd60a34a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:49:40.069 INFO 4486 --- [Collect obproxy1,70b191e5260217f1,527ca4537d026358] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:49:40.073 INFO 4486 --- [Collect obproxy1,70b191e5260217f1,527ca4537d026358] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:49:40.076 INFO 4486 --- [Collect obproxy1,70b191e5260217f1,527ca4537d026358] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:49:40.250 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:40.869 INFO 4486 --- [ocp-schedules-10,5c75c248a7b3a193,5cc21543f35f1608] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:49:40.994 INFO 4486 --- [prometheus-minute-schedule-1,1b9ba02638e3a698,5ed9a2be734a1c3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:40.995 INFO 4486 --- [prometheus-hour-schedule-1,07f8856bd0424710,26873a485c621c99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:49:41.002 INFO 4486 --- [prometheus-minute-schedule-1,1b9ba02638e3a698,5ed9a2be734a1c3c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:41.301 INFO 4486 --- [ocp-async-6,1b2a767e51e5ca1e,d4c53d13c5809aff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:41.511 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,99bd86f54f7e3565,0841829fcb0529b3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=99bd86f54f7e3565, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:41.511 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,99bd86f54f7e3565,0841829fcb0529b3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=99bd86f54f7e3565, duration=1 ms] 2024-10-16 13:49:41.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,77196fa6011e385b,c649c6c78d7e4816] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=77196fa6011e385b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:41.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,77196fa6011e385b,c649c6c78d7e4816] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=77196fa6011e385b, duration=1 ms] 2024-10-16 13:49:42.138 INFO 4486 --- [ocp-schedules-24,938a03cf97e908b9,c49542c4bbb6e097] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:49:43.010 INFO 4486 --- [ocp-async-3,ed664dfe7b987ef1,eb9708332269d62b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:49:43.252 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:43.301 INFO 4486 --- [ocp-async-1,9cc9ff643e7ed72f,24592a5106ac65af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:43.539 INFO 4486 --- [ocp-schedules-15,bfd7d5aee12af060,bc1df114627324fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:49:43.994 INFO 4486 --- [prometheus-minute-schedule-1,f26849482d2f63a4,7c5459dd22d16624] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:44.002 INFO 4486 --- [prometheus-minute-schedule-1,f26849482d2f63a4,7c5459dd22d16624] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:45.301 INFO 4486 --- [ocp-async-4,e213f57fb9c6c648,5b138c94002d6477] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:45.530 INFO 4486 --- [ocp-schedules-29,a9814045c1af7213,84d55a56e8f2f606] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:49:45.534 INFO 4486 --- [ocp-schedules-29,a9814045c1af7213,84d55a56e8f2f606] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:49:45.534 INFO 4486 --- [ocp-schedules-29,a9814045c1af7213,84d55a56e8f2f606] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:49:45.558 INFO 4486 --- [ocp-schedules-26,9b6a0778eac5cfff,88aab22f250109c3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:49:45.558 INFO 4486 --- [ocp-schedules-26,9b6a0778eac5cfff,88aab22f250109c3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:49:46.253 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:46.994 INFO 4486 --- [prometheus-minute-schedule-1,6af9eb5727f20a8e,aaa827b0a51681a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:46.999 INFO 4486 --- [prometheus-minute-schedule-1,6af9eb5727f20a8e,aaa827b0a51681a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:47.071 INFO 4486 --- [ocp-schedules-25,d7e4bc46336bdc65,07714ba7a0b854ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:49:47.171 INFO 4486 --- [ocp-schedules-21,4a2cb6904778a684,eab0953c55d51b56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:49:47.194 INFO 4486 --- [ocp-schedules-5,dca77fee9a524462,b470890f70f90df5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:49:47.302 INFO 4486 --- [ocp-async-6,9e0a543964a01c04,50a3cdf4c72f4050] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:47.831 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4397636fc3ee4096,aa21a175a4582dee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4397636fc3ee4096, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:47.831 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4397636fc3ee4096,aa21a175a4582dee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4397636fc3ee4096, duration=1 ms] 2024-10-16 13:49:48.375 INFO 4486 --- [ocp-schedules-27,8a163d5eb2a43602,8067e226e180cd52] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:49:48.406 WARN 4486 --- [scanner-13290743-1,8a163d5eb2a43602,79ef2c1de6725fec] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:49:48.469 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9395d12dc933728a,43ddf022d21e9b1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=9395d12dc933728a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:49:48.478 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9395d12dc933728a,43ddf022d21e9b1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=9395d12dc933728a, duration=29 ms] 2024-10-16 13:49:48.542 INFO 4486 --- [alarm-detect-8,8a163d5eb2a43602,ce801da3322ca781] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057755 ~ 1729057778 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057770, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057770, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057770=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:49:48.759 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:49:49.254 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:49.302 INFO 4486 --- [ocp-async-1,26e3601a46d8d03a,cac5a866e9bf71d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:49.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6e3888d6f39b5bf8,20007975a1bec30b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6e3888d6f39b5bf8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:49.987 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6e3888d6f39b5bf8,20007975a1bec30b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6e3888d6f39b5bf8, duration=1 ms] 2024-10-16 13:49:49.994 INFO 4486 --- [prometheus-minute-schedule-1,6da1fa1d6e10129d,5e968846ebb18f54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:50.001 INFO 4486 --- [prometheus-minute-schedule-1,6da1fa1d6e10129d,5e968846ebb18f54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:50.484 INFO 4486 --- [ocp-schedules-25,7323ec75f948cafc,10759bcedd27f28a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:49:50.484 INFO 4486 --- [ocp-schedules-25,7323ec75f948cafc,10759bcedd27f28a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:49:50.485 INFO 4486 --- [ocp-schedules-25,7323ec75f948cafc,10759bcedd27f28a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:49:50.485 INFO 4486 --- [ocp-schedules-25,7323ec75f948cafc,10759bcedd27f28a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:49:50.485 INFO 4486 --- [ocp-schedules-25,7323ec75f948cafc,10759bcedd27f28a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:49:50.485 INFO 4486 --- [ocp-schedules-25,7323ec75f948cafc,10759bcedd27f28a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:49:50.858 INFO 4486 --- [ocp-schedules-18,1d653cda2f3bdcfc,bb72b8bbfed7c213] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:49:50.858 INFO 4486 --- [ocp-schedules-18,1d653cda2f3bdcfc,bb72b8bbfed7c213] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:49:50.858 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:49:50.858 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:49:50.858 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:49:50.858 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:49:50.858 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:49:50.864 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:50.864 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:49:50.864 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:49:50.864 INFO 4486 --- [ocp-schedules-18,1d653cda2f3bdcfc,bb72b8bbfed7c213] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:49:50.864 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:49:50.864 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:49:50.866 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:49:50.866 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:49:50.866 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:50.866 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:49:50.866 INFO 4486 --- [ocp-schedules-18,1d653cda2f3bdcfc,bb72b8bbfed7c213] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:49:50.866 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:49:50.875 INFO 4486 --- [ocp-schedules-19,310c88c4e909d93c,5c52fdbb8f71f980] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:49:50.892 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:49:50.892 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:49:50.892 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:49:50.892 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:49:50.893 INFO 4486 --- [ocp-schedules-18,1d653cda2f3bdcfc,bb72b8bbfed7c213] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:49:50.893 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:49:50.905 INFO 4486 --- [ocp-schedules-18,1d653cda2f3bdcfc,bb72b8bbfed7c213] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:49:51.302 INFO 4486 --- [ocp-async-4,3200fb7ee673b1c7,7f963f926b2984aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:51.517 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9c60642bea36eff9,1ed2d5311999e5df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9c60642bea36eff9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:51.517 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9c60642bea36eff9,1ed2d5311999e5df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9c60642bea36eff9, duration=2 ms] 2024-10-16 13:49:51.831 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b6a79e6d819ff530,540c461603a64929] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b6a79e6d819ff530, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:51.831 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b6a79e6d819ff530,540c461603a64929] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b6a79e6d819ff530, duration=1 ms] 2024-10-16 13:49:52.255 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:52.994 INFO 4486 --- [prometheus-minute-schedule-1,65e1bb9d386ac882,d3560185cd02eca9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:53.001 INFO 4486 --- [prometheus-minute-schedule-1,65e1bb9d386ac882,d3560185cd02eca9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:53.011 INFO 4486 --- [ocp-async-9,6d9d33421512184e,6b3a3d103cbdc072] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:49:53.244 INFO 4486 --- [ocp-schedules-16,6d9c1f978215d05c,0eab68e2db15dfd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:49:53.244 INFO 4486 --- [ocp-schedules-16,3635d11577c62b4b,43470e04edc040a2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:49:53.246 INFO 4486 --- [ocp-schedules-16,3635d11577c62b4b,43470e04edc040a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:49:53.276 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.277 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:53.285 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.286 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:53.302 INFO 4486 --- [ocp-async-3,b8e19b174e85d7c1,79135ac9ee26148c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:53.312 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.312 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.313 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.313 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:49:53.313 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:53.341 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.342 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.342 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:53.343 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:49:53.344 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.347 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:49:53.348 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:53.357 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.357 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:53.367 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.376 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.376 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.377 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:53.378 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:49:53.383 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:49:53.389 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:49:53.391 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:49:53.418 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:49:53.431 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.433 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.434 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:49:53.474 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:49:53.474 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:49:53.493 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.495 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.496 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:49:53.499 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.503 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:49:53.503 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:49:53.506 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:53.514 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.514 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.515 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:49:53.526 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:49:53.526 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:49:53.533 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.533 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:53.560 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.560 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:53.585 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.585 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:53.613 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.613 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:49:53.639 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.639 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:49:53.668 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:49:53.695 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.696 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.697 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:49:53.701 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:49:53.706 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:49:53.710 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:49:53.712 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:49:53.737 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:49:53.880 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:49:53.911 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:49:53.912 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:49:53.913 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:49:53.949 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:49:53.949 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:49:55.256 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:55.302 INFO 4486 --- [ocp-async-1,e4be1b26b7e85824,06a96d6cd0b78a65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:55.994 INFO 4486 --- [prometheus-minute-schedule-1,750634ba73b5f238,d59b2b6a6c1f988f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:56.001 INFO 4486 --- [prometheus-minute-schedule-1,750634ba73b5f238,d59b2b6a6c1f988f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:57.302 INFO 4486 --- [ocp-async-9,b2903039a61db9de,d470b580f0612d66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:57.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8b774c67660b9109,ca2ef34a3e67c2c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8b774c67660b9109, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:57.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8b774c67660b9109,ca2ef34a3e67c2c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8b774c67660b9109, duration=2 ms] 2024-10-16 13:49:58.258 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:49:58.713 INFO 4486 --- [ocp-schedules-2,ff0d04ba8db5172b,e7daedf39451f375] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:49:58.750 WARN 4486 --- [scanner-13291094-1,ff0d04ba8db5172b,9550dbeade0f502c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:49:58.802 INFO 4486 --- [alarm-detect-2,ff0d04ba8db5172b,a5e5ed908bba4c80] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057770 ~ 1729057788 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057785, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057785, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057785=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:49:58.994 INFO 4486 --- [prometheus-minute-schedule-1,da8405a59f709466,16a8b78b583725a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057800, interval=60 2024-10-16 13:49:59.000 INFO 4486 --- [prometheus-minute-schedule-1,da8405a59f709466,16a8b78b583725a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:49:59.090 INFO 4486 --- [ocp-schedules-18,9099e25ea75ea5e7,784b11a0f7f53dab] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:49:59.303 INFO 4486 --- [ocp-async-5,123f152ed4117b32,89110669918f5c2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:49:59.992 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,466954c48919ed78,1ea1b4b5577e7d8e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=466954c48919ed78, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:49:59.992 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,466954c48919ed78,1ea1b4b5577e7d8e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=466954c48919ed78, duration=2 ms] 2024-10-16 13:50:00.883 INFO 4486 --- [ocp-schedules-2,0dbb09edbc6f1068,729e44a12fd38a90] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:50:01.154 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e5632d7b05742c0e,111b59967b75afc0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=e5632d7b05742c0e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:50:01.154 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,e5632d7b05742c0e,111b59967b75afc0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=e5632d7b05742c0e, duration=1 ms] 2024-10-16 13:50:01.180 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3319b324e0bde51d,58171f48a0b61906] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=3319b324e0bde51d, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 13:50:01.258 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:01.303 INFO 4486 --- [ocp-async-2,dd0bf1715f362e18,7a28e49c76cb8007] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:01.387 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3319b324e0bde51d,58171f48a0b61906] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=3319b324e0bde51d, duration=220 ms] 2024-10-16 13:50:01.522 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2b5b5fa1a7e3510a,d0a70113cf70e90d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2b5b5fa1a7e3510a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:01.522 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2b5b5fa1a7e3510a,d0a70113cf70e90d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2b5b5fa1a7e3510a, duration=1 ms] 2024-10-16 13:50:01.834 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,96f899df233dc5dc,f2f5f1ca1d8717fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=96f899df233dc5dc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:01.835 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,96f899df233dc5dc,f2f5f1ca1d8717fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=96f899df233dc5dc, duration=2 ms] 2024-10-16 13:50:01.994 INFO 4486 --- [prometheus-minute-schedule-1,289a27b6fb17466e,fcf88bdee77f2ff1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:02.001 INFO 4486 --- [prometheus-minute-schedule-1,289a27b6fb17466e,fcf88bdee77f2ff1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:02.041 INFO 4486 --- [ocp-schedules-25,feec6b186311cd05,6d300db0ef475bb2] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:50:02.041+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:50:02.045 INFO 4486 --- [ocp-schedules-25,feec6b186311cd05,6d300db0ef475bb2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:50:02.055 INFO 4486 --- [ocp-schedules-25,feec6b186311cd05,6d300db0ef475bb2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:50:02.066 INFO 4486 --- [ocp-schedules-25,feec6b186311cd05,6d300db0ef475bb2] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:50:03.011 INFO 4486 --- [ocp-async-7,f1d82a7c36b0c862,b7b3aa86d58956a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:50:03.303 INFO 4486 --- [ocp-async-8,51008c2c08642e5d,54b2267f73fb5df0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:04.260 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:04.749 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,f7fbca94e8f45175] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7933439, operation:EXECUTE, state: RUNNING 2024-10-16 13:50:04.764 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7933439, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266494, task_operation=execute, latest_execution_start_time=2024-10-16T13:50:04.741+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=7933439}, listMap={}}, executor=10.197.49.184 2024-10-16 13:50:04.770 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 13:50:04.771 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 13:50:04.773 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.774 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:50:04.776 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.776 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.777 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.778 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 13:50:04.797 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.798 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.799 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.799 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 13:50:04.814 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.815 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.816 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.816 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 13:50:04.833 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.833 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.834 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.835 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 13:50:04.850 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.850 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.851 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.852 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 13:50:04.859 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.860 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.861 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.861 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 13:50:04.867 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.868 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.869 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.869 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 13:50:04.875 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.876 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.876 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.877 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 13:50:04.885 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.886 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.887 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.888 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 13:50:04.894 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.895 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.896 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.896 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 13:50:04.903 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.904 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.904 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.905 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 13:50:04.918 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.919 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.919 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.920 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 13:50:04.941 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.942 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.943 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.943 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 13:50:04.949 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.950 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.950 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.951 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 13:50:04.958 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.958 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.959 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.959 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 13:50:04.967 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.967 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.968 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.969 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 13:50:04.975 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.975 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.976 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.976 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 13:50:04.981 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.982 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.983 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.984 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 13:50:04.989 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.990 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.990 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.991 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 13:50:04.994 INFO 4486 --- [prometheus-minute-schedule-1,94825829f84d4141,869711513b160971] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:04.998 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:04.998 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:04.999 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.000 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 13:50:05.000 INFO 4486 --- [prometheus-minute-schedule-1,94825829f84d4141,869711513b160971] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:05.006 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.007 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.008 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.009 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 13:50:05.015 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.015 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.016 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.017 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 13:50:05.022 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.022 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.023 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.024 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 13:50:05.029 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.030 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.030 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.031 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 13:50:05.037 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.037 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.039 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.040 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 13:50:05.045 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.045 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.046 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.047 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 13:50:05.052 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.053 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.054 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.055 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 13:50:05.060 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.061 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.061 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.062 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 13:50:05.067 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.067 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.068 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.069 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 13:50:05.076 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.077 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.078 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.078 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 13:50:05.083 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.083 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.084 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.085 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 13:50:05.091 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.091 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.092 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.093 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 13:50:05.103 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.104 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.104 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.105 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 13:50:05.110 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.111 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.111 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.112 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 13:50:05.121 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.122 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.123 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.124 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 13:50:05.132 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.132 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.133 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.134 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 13:50:05.139 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.140 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.140 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.141 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 13:50:05.146 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.147 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.147 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.148 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 13:50:05.152 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.153 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.153 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.154 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 13:50:05.158 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.158 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.159 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.159 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 13:50:05.163 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.164 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.165 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.165 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 13:50:05.169 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.169 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.170 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.171 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 13:50:05.182 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.182 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.183 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.183 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 13:50:05.206 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.206 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.207 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.207 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 13:50:05.214 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.215 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.216 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.216 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 13:50:05.224 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.224 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.225 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.226 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 13:50:05.233 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.233 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.234 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.234 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 13:50:05.239 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.240 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.240 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.241 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 13:50:05.245 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.246 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.246 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.247 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 13:50:05.251 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 13:50:05.251 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 13:50:05.259 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.259 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:50:05.260 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.261 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.262 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.263 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 13:50:05.270 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.270 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:50:05.271 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:05.272 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 13:50:05.277 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 13:50:05.285 INFO 4486 --- [subtask-executor32,2233273729644a0c,60cc1eb53936ec67] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7933439, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 13:50:05.303 INFO 4486 --- [ocp-async-6,a298b2362a07b789,575436b5bca35a9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:07.066 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:50:07.066 INFO 4486 --- [ocp-schedules-7,9ad826a2d77f31f6,38384181d20e9d39] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:50:07.068 INFO 4486 --- [ocp-schedules-7,9ad826a2d77f31f6,38384181d20e9d39] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:07.081 INFO 4486 --- [ocp-schedules-11,9f3dd5fb6df1d0d1,c37e74500930c867] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:50:07.082 INFO 4486 --- [ocp-schedules-11,9f3dd5fb6df1d0d1,c37e74500930c867] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:50:07.089 INFO 4486 --- [ocp-schedules-4,bd630b2c81dc9841,364fa860ea6bf175] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:50:07.089 INFO 4486 --- [ocp-schedules-14,040effbd6b96493a,ac5b95f11c88407a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:50:07.089 INFO 4486 --- [ocp-schedules-6,c05c4b6cecdba4a2,ad0e0de3c24deed4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:50:07.091 INFO 4486 --- [host-check-agent-status1,bd630b2c81dc9841,b0b35d7620949ab0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:50:07.091 INFO 4486 --- [host-check-agent-status2,bd630b2c81dc9841,a34ba1fd82fdc864] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:50:07.091 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:50:07.091 INFO 4486 --- [host-check-clockdiff1,c05c4b6cecdba4a2,fca681726d005d63] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:50:07.091 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:50:07.092 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:50:07.092 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:50:07.092 INFO 4486 --- [host-check-agent-status3,bd630b2c81dc9841,f9fb8664727985db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:50:07.093 INFO 4486 --- [host-check-agent-status4,bd630b2c81dc9841,0362e7ba94a64d3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:50:07.094 INFO 4486 --- [host-check-clockdiff2,c05c4b6cecdba4a2,a304c8ced7c15e79] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:50:07.094 INFO 4486 --- [host-check-clockdiff3,c05c4b6cecdba4a2,e69c473b045d4aef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:50:07.095 INFO 4486 --- [host-check-clockdiff4,c05c4b6cecdba4a2,9b10081e5ca5f97b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:50:07.097 INFO 4486 --- [host-check-agent-status2,bd630b2c81dc9841,a34ba1fd82fdc864] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:50:07.097 INFO 4486 --- [host-check-agent-status1,bd630b2c81dc9841,b0b35d7620949ab0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:07.099 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:07.099 INFO 4486 --- [host-check-agent-status3,bd630b2c81dc9841,f9fb8664727985db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:50:07.099 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:50:07.101 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:50:07.101 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:50:07.101 INFO 4486 --- [host-check-agent-status2,bd630b2c81dc9841,a34ba1fd82fdc864] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:50:07.101 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:07.101 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:50:07.101 INFO 4486 --- [host-check-agent-status1,bd630b2c81dc9841,b0b35d7620949ab0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:07.112 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:50:07.112 INFO 4486 --- [host-check-agent-status4,bd630b2c81dc9841,0362e7ba94a64d3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:50:07.112 INFO 4486 --- [host-check-agent-status3,bd630b2c81dc9841,f9fb8664727985db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:50:07.112 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:50:07.117 INFO 4486 --- [host-check-clockdiff4,c05c4b6cecdba4a2,9b10081e5ca5f97b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057807 4 4 , error= 2024-10-16 13:50:07.120 INFO 4486 --- [host-check-clockdiff2,c05c4b6cecdba4a2,a304c8ced7c15e79] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057807 4 4 , error= 2024-10-16 13:50:07.120 INFO 4486 --- [host-check-clockdiff3,c05c4b6cecdba4a2,e69c473b045d4aef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057807 4 4 , error= 2024-10-16 13:50:07.120 INFO 4486 --- [host-check-clockdiff1,c05c4b6cecdba4a2,fca681726d005d63] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057807 0 0 , error= 2024-10-16 13:50:07.120 INFO 4486 --- [host-check-agent-status4,bd630b2c81dc9841,0362e7ba94a64d3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:50:07.134 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.135 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:07.136 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:50:07.147 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.147 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:50:07.148 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:07.148 INFO 4486 --- [host-check-agent-status1,bd630b2c81dc9841,b0b35d7620949ab0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:50:07.148 INFO 4486 --- [host-check-agent-status3,bd630b2c81dc9841,f9fb8664727985db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:50:07.150 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:50:07.151 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:50:07.150105692+08:00 2024-10-16 13:50:07.152 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:50:07.151+08:00 2024-10-16 13:50:07.152 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:50:07.156349115+08:00 2024-10-16 13:50:07.152 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:50:07.153 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:50:07.153 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:50:07.152+08:00 2024-10-16 13:50:07.153 INFO 4486 --- [host-check-agent-status3,bd630b2c81dc9841,f9fb8664727985db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:50:07.153 INFO 4486 --- [host-check-agent-status3,bd630b2c81dc9841,f9fb8664727985db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:50:07.153 INFO 4486 --- [host-check-agent-status3,bd630b2c81dc9841,f9fb8664727985db] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:50:07.154 INFO 4486 --- [host-check-agent-status4,bd630b2c81dc9841,0362e7ba94a64d3a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:50:07.154 INFO 4486 --- [host-check-agent-status1,bd630b2c81dc9841,b0b35d7620949ab0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:50:07.154 INFO 4486 --- [host-check-agent-status1,bd630b2c81dc9841,b0b35d7620949ab0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:50:07.154 INFO 4486 --- [host-check-agent-status1,bd630b2c81dc9841,b0b35d7620949ab0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:50:07.154 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:50:07.155 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:50:07.151+08:00 and status AVAILABLE 2024-10-16 13:50:07.155 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:50:07.152+08:00 and status AVAILABLE 2024-10-16 13:50:07.155 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:50:07.158931591+08:00 2024-10-16 13:50:07.156 INFO 4486 --- [host-check-agent-status2,bd630b2c81dc9841,a34ba1fd82fdc864] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:50:07.156 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:50:07.155+08:00 2024-10-16 13:50:07.157 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:50:07.161007378+08:00 2024-10-16 13:50:07.157 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:50:07.158 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:50:07.157+08:00 2024-10-16 13:50:07.159 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:50:07.159 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:50:07.155+08:00 and status AVAILABLE 2024-10-16 13:50:07.160 INFO 4486 --- [host-check-agent-status4,bd630b2c81dc9841,0362e7ba94a64d3a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:50:07.160 INFO 4486 --- [host-check-agent-status4,bd630b2c81dc9841,0362e7ba94a64d3a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:50:07.160 INFO 4486 --- [host-check-agent-status4,bd630b2c81dc9841,0362e7ba94a64d3a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:50:07.160 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:50:07.157+08:00 and status AVAILABLE 2024-10-16 13:50:07.162 INFO 4486 --- [host-check-agent-status2,bd630b2c81dc9841,a34ba1fd82fdc864] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:50:07.162 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.162 INFO 4486 --- [host-check-agent-status2,bd630b2c81dc9841,a34ba1fd82fdc864] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:50:07.162 INFO 4486 --- [host-check-agent-status2,bd630b2c81dc9841,a34ba1fd82fdc864] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:50:07.163 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:50:07.163 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:50:07.166 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:07.167 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:50:07.168 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:50:07.168 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:07.169 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:50:07.172 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:50:07.174 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:50:07.174 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:50:07.176 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.176 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:50:07.177 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:07.189 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.189 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:07.193 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:07.198 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:07.199 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.199 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:07.202 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.202 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:07.202 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:07.202 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:50:07.216 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:07.218 INFO 4486 --- [ocp-schedules-2,1ba8867277036f17,d5158bb51b51b533] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:50:07.223 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:50:07.234 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:50:07.237 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:50:07.247 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:50:07.249 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:50:07.251 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:50:07.253 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:07.259 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:50:07.261 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:07.264 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:50:07.269 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:07.277 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:07.280 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:07.286 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:50:07.297 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:50:07.302 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:50:07.303 INFO 4486 --- [ocp-async-3,5f0775f4d58b74dd,217856d0f9833957] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:07.314 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:07.315 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:50:07.316 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:50:07.325 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:50:07.334 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:07.337 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:50:07.340 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:07.343 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:50:07.347 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:50:07.354 INFO 4486 --- [host-check-host-status3,040effbd6b96493a,6ba5b8a93e373ec0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:50:07.364 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:07.368 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:50:07.380 INFO 4486 --- [host-check-host-status4,040effbd6b96493a,8332eeefd45b5867] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:50:07.381 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:50:07.392 INFO 4486 --- [host-check-host-status2,040effbd6b96493a,b8a323e20fcda54d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:50:07.430 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:50:07.441 INFO 4486 --- [host-check-host-status1,040effbd6b96493a,9bdd752a6a0aeb35] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:50:07.848 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,dc683323000ce655,4ef362da58ec23da] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dc683323000ce655, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:07.848 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,dc683323000ce655,4ef362da58ec23da] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dc683323000ce655, duration=2 ms] 2024-10-16 13:50:07.994 INFO 4486 --- [prometheus-minute-schedule-1,b50cff6b0b9a2bfd,4b6e62ee265109bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:07.999 INFO 4486 --- [prometheus-minute-schedule-1,b50cff6b0b9a2bfd,4b6e62ee265109bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:08.877 INFO 4486 --- [ocp-schedules-4,0e54ece01c292d34,4c81287e5f6709ae] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:50:08.917 WARN 4486 --- [scanner-13291350-1,0e54ece01c292d34,af961dbc1a9d30ee] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:50:09.304 INFO 4486 --- [ocp-async-1,30ba23edf84b5208,7ff52a070c1be289] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:09.386 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:50:09.387 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:50:09.395 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:50:09.395 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:50:09.395 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:50:09.395 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:50:09.401 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:50:09.402 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:50:09.402 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:50:09.402 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:50:09.464 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:50:09.473 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:50:09.473 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:50:09.473 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:50:09.473 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:50:09.473 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:50:09.480 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:50:09.998 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b06df823df3691e2,3a15710008eceb87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b06df823df3691e2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:09.998 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b06df823df3691e2,3a15710008eceb87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b06df823df3691e2, duration=2 ms] 2024-10-16 13:50:10.005 INFO 4486 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:50:10.005 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:50:10.006 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:50:10.030 INFO 4486 --- [Collect obproxy1,2af83eaac8073607,d97c640cac21ace5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:50:10.030 INFO 4486 --- [Collect obproxy2,a03edec23e662800,d487cbf69f0610ef] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:50:10.031 INFO 4486 --- [Collect obproxy2,a03edec23e662800,d487cbf69f0610ef] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:10.031 INFO 4486 --- [Collect obproxy1,2af83eaac8073607,d97c640cac21ace5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:10.031 INFO 4486 --- [Check obproxy2,697297d3d404bdaf,a4643dcb6968bd01] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:50:10.031 INFO 4486 --- [Check obproxy2,697297d3d404bdaf,a4643dcb6968bd01] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:10.032 INFO 4486 --- [Check obproxy1,e530095f65358630,071fc81aed9c053b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:50:10.032 INFO 4486 --- [Check obproxy1,e530095f65358630,071fc81aed9c053b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:10.054 INFO 4486 --- [Collect obproxy2,a03edec23e662800,d487cbf69f0610ef] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:50:10.055 INFO 4486 --- [Collect obproxy1,2af83eaac8073607,d97c640cac21ace5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:50:10.055 INFO 4486 --- [Check obproxy2,697297d3d404bdaf,a4643dcb6968bd01] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:50:10.055 INFO 4486 --- [Check obproxy2,697297d3d404bdaf,a4643dcb6968bd01] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:50:10.057 INFO 4486 --- [Collect obproxy1,2af83eaac8073607,d97c640cac21ace5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:50:10.058 INFO 4486 --- [Check obproxy1,e530095f65358630,071fc81aed9c053b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:50:10.058 INFO 4486 --- [Check obproxy1,e530095f65358630,071fc81aed9c053b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:50:10.059 INFO 4486 --- [Collect obproxy2,a03edec23e662800,d487cbf69f0610ef] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:50:10.062 INFO 4486 --- [Collect obproxy1,2af83eaac8073607,d97c640cac21ace5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:50:10.063 INFO 4486 --- [Collect obproxy2,a03edec23e662800,d487cbf69f0610ef] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:50:10.262 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:10.524 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.122339442S 2024-10-16 13:50:10.524 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:50:10.592 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:50:10.592 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:50:10.620 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:50:10.712 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:50:10.712 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:50:10.712 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:50:10.712 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:50:10.712 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:50:10.712 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:50:10.720 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:50:10.837 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] o.s.boot.SpringApplication : Started application in 1.442 seconds (JVM running for 78713.888) 2024-10-16 13:50:10.838 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.442730866S 2024-10-16 13:50:10.890 INFO 4486 --- [ocp-schedules-10,fee228bb3ca9b0a6,61f065a5cd6b4cdc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:50:10.980 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:50:10.994 INFO 4486 --- [prometheus-minute-schedule-1,8d90166ee638b037,7b6b16ad049ed6c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:11.001 INFO 4486 --- [prometheus-minute-schedule-1,8d90166ee638b037,7b6b16ad049ed6c7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:50:11.193 INFO 4486 --- [ocp-schedules-15,b7f332a884c1b9a1,8e578c6c63103731] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:50:11.304 INFO 4486 --- [ocp-async-10,3fa276bc56f391a0,6d093f3b70dcb5c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:11.527 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dd0c11e4fb911490,9c596f6282b7e8f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dd0c11e4fb911490, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:11.527 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,dd0c11e4fb911490,9c596f6282b7e8f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dd0c11e4fb911490, duration=2 ms] 2024-10-16 13:50:11.838 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c99150e01209ce21,f9cb9edd7c6ad346] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c99150e01209ce21, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:11.838 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c99150e01209ce21,f9cb9edd7c6ad346] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c99150e01209ce21, duration=1 ms] 2024-10-16 13:50:13.011 INFO 4486 --- [ocp-async-8,ffa788838f0fb88e,930ba11135ecdd4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:50:13.263 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:13.304 INFO 4486 --- [ocp-async-7,28fac8457c1059fd,e97f7edd14df6f9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:13.994 INFO 4486 --- [prometheus-minute-schedule-1,e0245a5012e99ba3,6cf624dd1811b4a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:14.002 INFO 4486 --- [prometheus-minute-schedule-1,e0245a5012e99ba3,6cf624dd1811b4a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:14.125 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057740, nodeCount=450, affectRows=450 2024-10-16 13:50:14.128 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057740, nodeCount=449, validCount=449 2024-10-16 13:50:14.147 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057740, nodeCount=1000, affectRows=1000 2024-10-16 13:50:14.149 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057740, nodeCount=1000, affectRows=1000 2024-10-16 13:50:14.151 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057740, nodeCount=1000, validCount=1000 2024-10-16 13:50:14.153 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057740, nodeCount=998, validCount=998 2024-10-16 13:50:14.160 INFO 4486 --- [ocp-schedules-29,09bdcc0dd171eae2,55ad96a44cf0a834] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 13:50:14.160 INFO 4486 --- [ocp-schedules-29,09bdcc0dd171eae2,55ad96a44cf0a834] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 13:50:14.162 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057740, nodeCount=1000, affectRows=1000 2024-10-16 13:50:14.167 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057740, nodeCount=1000, validCount=1000 2024-10-16 13:50:14.199 INFO 4486 --- [pool-281-thread-2,09bdcc0dd171eae2,23ecedf358c33dde] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:50:14.199 INFO 4486 --- [pool-281-thread-1,09bdcc0dd171eae2,355f7f0c566aa118] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:50:14.200 INFO 4486 --- [pool-281-thread-3,09bdcc0dd171eae2,41673e5d6eb92c00] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:50:14.229 INFO 4486 --- [pool-281-thread-1,09bdcc0dd171eae2,355f7f0c566aa118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:14.231 INFO 4486 --- [pool-281-thread-2,09bdcc0dd171eae2,23ecedf358c33dde] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:14.233 INFO 4486 --- [pool-281-thread-1,09bdcc0dd171eae2,355f7f0c566aa118] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057814000000] 2024-10-16 13:50:14.235 INFO 4486 --- [pool-281-thread-2,09bdcc0dd171eae2,23ecedf358c33dde] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057814000000] 2024-10-16 13:50:14.239 INFO 4486 --- [pool-281-thread-3,09bdcc0dd171eae2,41673e5d6eb92c00] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:14.243 INFO 4486 --- [pool-281-thread-3,09bdcc0dd171eae2,41673e5d6eb92c00] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729057814000000] 2024-10-16 13:50:14.515 WARN 4486 --- [pool-281-thread-3,09bdcc0dd171eae2,41673e5d6eb92c00] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 13:50:14.515 WARN 4486 --- [pool-281-thread-3,09bdcc0dd171eae2,41673e5d6eb92c00] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 13:50:14.517 INFO 4486 --- [pool-281-thread-3,09bdcc0dd171eae2,41673e5d6eb92c00] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:50:14.865 WARN 4486 --- [pool-281-thread-1,09bdcc0dd171eae2,355f7f0c566aa118] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 13:50:14.865 WARN 4486 --- [pool-281-thread-1,09bdcc0dd171eae2,355f7f0c566aa118] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1009 2024-10-16 13:50:14.868 INFO 4486 --- [pool-281-thread-1,09bdcc0dd171eae2,355f7f0c566aa118] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:50:14.969 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057740, nodeCount=50, affectRows=50 2024-10-16 13:50:14.970 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057740, nodeCount=75, affectRows=75 2024-10-16 13:50:14.977 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057740, nodeCount=75, affectRows=75 2024-10-16 13:50:14.992 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057740, nodeCount=277, affectRows=277 2024-10-16 13:50:15.057 INFO 4486 --- [metric-archive-async-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057740, nodeCount=50, validCount=50 2024-10-16 13:50:15.091 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057740, nodeCount=75, validCount=75 2024-10-16 13:50:15.103 INFO 4486 --- [metric-archive-async-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057740, nodeCount=75, validCount=75 2024-10-16 13:50:15.216 WARN 4486 --- [pool-281-thread-2,09bdcc0dd171eae2,23ecedf358c33dde] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 13:50:15.217 INFO 4486 --- [pool-281-thread-2,09bdcc0dd171eae2,23ecedf358c33dde] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:50:15.219 INFO 4486 --- [ocp-schedules-29,09bdcc0dd171eae2,55ad96a44cf0a834] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 13:50:15.304 INFO 4486 --- [ocp-async-1,7158cd8d051a9208,fd8c3458d047c4cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:15.393 INFO 4486 --- [metric-archive-async-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057740, nodeCount=277, validCount=277 2024-10-16 13:50:16.264 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:16.994 INFO 4486 --- [prometheus-minute-schedule-1,936207a11d250166,08a4cb42f6fbfed4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:17.000 INFO 4486 --- [prometheus-minute-schedule-1,936207a11d250166,08a4cb42f6fbfed4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:17.070 INFO 4486 --- [ocp-schedules-7,9ad826a2d77f31f6,38384181d20e9d39] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:50:17.153 INFO 4486 --- [ocp-schedules-11,9f3dd5fb6df1d0d1,c37e74500930c867] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:50:17.197 INFO 4486 --- [ocp-schedules-25,fbeaf40b977390fa,bae7128e8b752ea0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:50:17.304 INFO 4486 --- [ocp-async-8,4d8db5ef979ab961,565bc41b5629b760] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:17.854 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc0084542476c800,3afc9eaee52eba1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cc0084542476c800, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:17.854 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc0084542476c800,3afc9eaee52eba1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cc0084542476c800, duration=1 ms] 2024-10-16 13:50:19.043 INFO 4486 --- [ocp-schedules-5,cadb79a85d38ac89,905325c6773985dd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:50:19.064 WARN 4486 --- [scanner-13292468-1,cadb79a85d38ac89,27d4cf11995d9608] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:50:19.101 INFO 4486 --- [ocp-schedules-13,c311e349d5cec710,41f9f1668fcbe301] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:50:19.139 INFO 4486 --- [alarm-detect-17,cadb79a85d38ac89,122997a5d1904d84] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057785 ~ 1729057809 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057800, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057800, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057800=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:50:19.265 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:19.305 INFO 4486 --- [ocp-async-6,bb7cc6b3c6143703,e53e30199bedbdaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:19.994 INFO 4486 --- [prometheus-minute-schedule-1,62da198297af1cbb,b8f750473c477938] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:20.000 INFO 4486 --- [prometheus-minute-schedule-1,62da198297af1cbb,b8f750473c477938] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:20.003 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,361d27d6b2223ccf,fb9622b68d137b9b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=361d27d6b2223ccf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:20.003 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,361d27d6b2223ccf,fb9622b68d137b9b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=361d27d6b2223ccf, duration=2 ms] 2024-10-16 13:50:20.812 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:50:20.895 INFO 4486 --- [ocp-schedules-15,14a6399235f8cc34,c9151d8b6e9f0eb1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:50:20.996 INFO 4486 --- [prometheus-cache-monitor1,41b862f822a11476,d2e47f9d666adbf8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:50:20.996 INFO 4486 --- [prometheus-cache-monitor1,41b862f822a11476,d2e47f9d666adbf8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:50:21.305 INFO 4486 --- [ocp-async-2,0fa4b64ddea4febd,0f2570117f1f03b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:21.538 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,97177e81c572a55a,0fb72ec91a532679] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=97177e81c572a55a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:21.538 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,97177e81c572a55a,0fb72ec91a532679] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=97177e81c572a55a, duration=1 ms] 2024-10-16 13:50:21.846 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,078d17e57ee2650a,14719b36e47eb59b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=078d17e57ee2650a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:21.846 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,078d17e57ee2650a,14719b36e47eb59b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=078d17e57ee2650a, duration=1 ms] 2024-10-16 13:50:21.879 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d3f03891d2f2c3c7,0bfece38432af802] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=d3f03891d2f2c3c7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:50:21.880 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d3f03891d2f2c3c7,0bfece38432af802] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=d3f03891d2f2c3c7, duration=2 ms] 2024-10-16 13:50:22.266 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:22.994 INFO 4486 --- [prometheus-minute-schedule-1,91f52b16e57866b6,a7d1b46f3e63da8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:23.000 INFO 4486 --- [prometheus-minute-schedule-1,91f52b16e57866b6,a7d1b46f3e63da8e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:23.011 INFO 4486 --- [ocp-async-7,1e7c52481d3153ee,a5affada407390d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:50:23.305 INFO 4486 --- [ocp-async-9,e4f65cd3a12f6ddd,e00f74448608b06f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:23.526 INFO 4486 --- [ocp-schedules-23,d16dd5e7b3281930,6ab9074bb0c644ce] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:50:23.950 INFO 4486 --- [ocp-schedules-16,3635d11577c62b4b,43470e04edc040a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:50:23.950 INFO 4486 --- [ocp-schedules-16,1ece14fd881c8303,94d47b92ba66d740] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:50:23.953 INFO 4486 --- [ocp-schedules-16,1ece14fd881c8303,94d47b92ba66d740] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:23.989 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:23.990 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:23.995 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:23.996 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:24.027 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.028 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:24.035 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.040 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.041 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:50:24.076 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.077 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.077 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:24.079 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:50:24.083 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.086 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:50:24.088 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:24.102 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.102 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:24.116 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.125 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.125 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.126 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:24.127 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:50:24.130 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.131 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:50:24.132 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.133 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:50:24.136 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:24.138 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:24.161 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:50:24.161 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:50:24.161 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:50:24.179 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.181 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.182 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:24.185 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.188 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:50:24.192 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:24.216 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.217 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:24.234 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:50:24.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:24.243 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.248 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.249 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:50:24.257 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:50:24.258 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:50:24.267 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.267 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:24.294 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.294 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:24.317 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.317 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:24.344 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:24.371 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.372 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.373 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:24.378 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:50:24.383 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:50:24.387 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:50:24.388 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:50:24.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:50:24.564 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:50:24.590 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:24.592 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:24.593 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:50:24.624 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:50:24.624 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:50:24.833 INFO 4486 --- [ocp-schedules-18,3be219a5d415b347,1e4d694766a90d01] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:50:25.267 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:25.306 INFO 4486 --- [ocp-async-5,525eaec57290e4c5,b0e0e2193c8f2d65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:25.877 INFO 4486 --- [ocp-schedules-17,dfe1ec7d5ec0b930,89fef68cbfab2c87] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:50:25.877 INFO 4486 --- [ocp-schedules-17,dfe1ec7d5ec0b930,89fef68cbfab2c87] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:50:25.916 INFO 4486 --- [Collect all tenants major compactions1,dfe1ec7d5ec0b930,bface93fd0226dd2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:50:25.916 INFO 4486 --- [Collect all tenants major compactions3,dfe1ec7d5ec0b930,0dafadc710a99c01] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:50:25.917 INFO 4486 --- [Collect all tenants major compactions2,dfe1ec7d5ec0b930,0c166995a5b6a278] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:50:25.917 INFO 4486 --- [Collect all tenants major compactions4,dfe1ec7d5ec0b930,7cdaa27992eae213] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:50:25.917 INFO 4486 --- [Collect all tenants major compactions5,dfe1ec7d5ec0b930,0bde5f2e7b87ffaa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:50:25.934 INFO 4486 --- [Collect all tenants major compactions1,dfe1ec7d5ec0b930,bface93fd0226dd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:25.936 INFO 4486 --- [Collect all tenants major compactions1,dfe1ec7d5ec0b930,bface93fd0226dd2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:50:25.937 INFO 4486 --- [Collect all tenants major compactions2,dfe1ec7d5ec0b930,0c166995a5b6a278] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:25.937 INFO 4486 --- [Collect all tenants major compactions3,dfe1ec7d5ec0b930,0dafadc710a99c01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:25.938 INFO 4486 --- [Collect all tenants major compactions2,dfe1ec7d5ec0b930,0c166995a5b6a278] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:50:25.939 INFO 4486 --- [Collect all tenants major compactions3,dfe1ec7d5ec0b930,0dafadc710a99c01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:50:25.940 INFO 4486 --- [Collect all tenants major compactions1,dfe1ec7d5ec0b930,bface93fd0226dd2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:50:25.942 INFO 4486 --- [Collect all tenants major compactions2,dfe1ec7d5ec0b930,0c166995a5b6a278] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:50:25.943 INFO 4486 --- [Collect all tenants major compactions3,dfe1ec7d5ec0b930,0dafadc710a99c01] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:50:25.944 INFO 4486 --- [Collect all tenants major compactions1,dfe1ec7d5ec0b930,bface93fd0226dd2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:50:25.949 INFO 4486 --- [Collect all tenants major compactions2,dfe1ec7d5ec0b930,0c166995a5b6a278] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:50:25.950 INFO 4486 --- [Collect all tenants major compactions3,dfe1ec7d5ec0b930,0dafadc710a99c01] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:50:25.955 INFO 4486 --- [Collect all tenants major compactions5,dfe1ec7d5ec0b930,0bde5f2e7b87ffaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:25.955 INFO 4486 --- [Collect all tenants major compactions4,dfe1ec7d5ec0b930,7cdaa27992eae213] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:25.957 INFO 4486 --- [Collect all tenants major compactions4,dfe1ec7d5ec0b930,7cdaa27992eae213] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:50:25.958 INFO 4486 --- [Collect all tenants major compactions5,dfe1ec7d5ec0b930,0bde5f2e7b87ffaa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:50:25.960 INFO 4486 --- [Collect all tenants major compactions4,dfe1ec7d5ec0b930,7cdaa27992eae213] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:50:25.960 INFO 4486 --- [Collect all tenants major compactions4,dfe1ec7d5ec0b930,7cdaa27992eae213] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:50:25.963 INFO 4486 --- [Collect all tenants major compactions5,dfe1ec7d5ec0b930,0bde5f2e7b87ffaa] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:50:25.963 INFO 4486 --- [Collect all tenants major compactions5,dfe1ec7d5ec0b930,0bde5f2e7b87ffaa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:50:25.964 INFO 4486 --- [ocp-schedules-17,dfe1ec7d5ec0b930,89fef68cbfab2c87] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:50:25.994 INFO 4486 --- [prometheus-minute-schedule-1,1b73fd1ccfa648b7,c297a6b10f7e370b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:26.002 INFO 4486 --- [prometheus-minute-schedule-1,1b73fd1ccfa648b7,c297a6b10f7e370b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:27.044 INFO 4486 --- [ocp-schedules-21,1f8dec7b2804b4c5,e6d21cc30e1a30d2] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:50:27.046 INFO 4486 --- [ocp-schedules-21,1f8dec7b2804b4c5,e6d21cc30e1a30d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:27.084 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:50:27.084 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.089 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:50:27.102 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.103 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:50:27.103 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:50:27.104 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.108 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:50:27.112 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.112 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:50:27.121 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.122 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:50:27.123 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.124 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:50:27.130 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.130 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:50:27.135 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.136 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:50:27.137 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:50:27.138 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.138 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.139 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:27.142 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:50:27.147 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.148 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:27.148 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.149 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:27.154 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.155 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:50:27.155 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.156 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:50:27.159 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.160 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:27.164 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.164 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:50:27.166 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.167 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:50:27.170 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.170 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:50:27.181 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.182 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.182 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:50:27.183 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:50:27.184 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.185 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.186 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.187 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.187 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.188 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:50:27.188 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.190 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.191 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.192 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.193 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.194 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.194 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.195 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.195 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.196 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.197 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.198 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.198 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.198 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:50:27.198 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.199 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.200 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.200 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.201 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.202 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.202 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.203 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.203 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:50:27.206 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.209 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.209 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.210 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:50:27.210 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:50:27.211 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.212 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.212 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.213 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:27.215 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.216 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.220 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.221 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.222 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.223 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:27.223 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.225 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.225 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:27.226 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.226 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:50:27.227 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.229 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.231 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.232 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.232 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.233 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:27.235 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:50:27.236 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.239 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.239 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:50:27.239 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:50:27.240 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.242 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:50:27.245 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:50:27.261 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.262 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.266 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.267 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.270 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.270 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:27.271 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.272 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.273 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:50:27.275 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.276 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.278 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.279 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.282 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.283 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.283 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.283 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:50:27.284 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.284 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:50:27.284 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.284 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:50:27.287 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.289 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:50:27.289 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:50:27.289 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:50:27.292 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.294 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:50:27.297 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.299 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:50:27.301 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:50:27.305 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:50:27.306 INFO 4486 --- [ocp-async-7,dbd3a0e494e8efe0,1b1d3220507640fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:27.365 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:27.367 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:27.368 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:50:27.377 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.377 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:50:27.377 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.377 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:50:27.377 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:50:27.377 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:50:27.381 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:50:27.381 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:50:27.492 INFO 4486 --- [ocp-schedules-22,1994636310391cb8,d5310cf24dc3e88b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:50:27.491+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:50:27.493 INFO 4486 --- [ocp-schedules-22,1994636310391cb8,d5310cf24dc3e88b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:50:27.494 INFO 4486 --- [ocp-schedules-22,1994636310391cb8,d5310cf24dc3e88b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:50:27.495 INFO 4486 --- [ocp-schedules-22,1994636310391cb8,d5310cf24dc3e88b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:50:27.859 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3125c1cd7db6f3ff,ae8704bd110e5075] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3125c1cd7db6f3ff, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:27.860 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3125c1cd7db6f3ff,ae8704bd110e5075] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3125c1cd7db6f3ff, duration=3 ms] 2024-10-16 13:50:28.268 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:28.994 INFO 4486 --- [prometheus-minute-schedule-1,9a485423ab43007c,17389205d85fc580] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:29.000 INFO 4486 --- [prometheus-minute-schedule-1,9a485423ab43007c,17389205d85fc580] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:29.204 INFO 4486 --- [ocp-schedules-14,2cf57dae2a15c269,f3ed5be8d13d234d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:50:29.243 WARN 4486 --- [scanner-13292719-1,2cf57dae2a15c269,d73c347b3bf12587] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:50:29.303 INFO 4486 --- [alarm-detect-5,2cf57dae2a15c269,45bf712304d969a2] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057800 ~ 1729057819 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057815, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057815, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057815=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:50:29.306 INFO 4486 --- [ocp-async-10,50c97630e483c034,4ec5b2b43f4fa4ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:30.007 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c268e44469c08613,2ce0f5c9338923ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c268e44469c08613, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:30.007 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c268e44469c08613,2ce0f5c9338923ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c268e44469c08613, duration=1 ms] 2024-10-16 13:50:30.102 INFO 4486 --- [prometheus-second-schedule-1,0a9b9af5dc1498dc,b22cdfc453fe0815] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:50:30.901 INFO 4486 --- [ocp-schedules-27,a5b4b5f3b8377f83,86511c87ca56bb07] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:50:31.269 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:31.306 INFO 4486 --- [ocp-async-3,dade7f16f99ccb9c,59b860c6c451af8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:31.543 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,caceb127bf8a39ff,1ba99c4680143efb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=caceb127bf8a39ff, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:31.543 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,caceb127bf8a39ff,1ba99c4680143efb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=caceb127bf8a39ff, duration=1 ms] 2024-10-16 13:50:31.851 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d578ca67f3b2adc3,660d1e13909fa8ef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d578ca67f3b2adc3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:31.851 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d578ca67f3b2adc3,660d1e13909fa8ef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d578ca67f3b2adc3, duration=1 ms] 2024-10-16 13:50:31.994 INFO 4486 --- [prometheus-minute-schedule-1,3d76eb96bdfa9cd9,2889082aa60b075a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:32.000 INFO 4486 --- [prometheus-minute-schedule-1,3d76eb96bdfa9cd9,2889082aa60b075a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:32.294 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6362203f79f1ee7f,a2bb95f8536beb7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=6362203f79f1ee7f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:50:32.295 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6362203f79f1ee7f,a2bb95f8536beb7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=6362203f79f1ee7f, duration=30 ms] 2024-10-16 13:50:33.011 INFO 4486 --- [ocp-async-8,368d8666d2872a90,d75fc55140cceb7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:50:33.306 INFO 4486 --- [ocp-async-7,20e0fc3201e62a3f,f04d51d481109d1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:34.270 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:34.994 INFO 4486 --- [prometheus-minute-schedule-1,8b2adaab719ab2b0,6ebb698ac570b334] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:35.001 INFO 4486 --- [prometheus-minute-schedule-1,8b2adaab719ab2b0,6ebb698ac570b334] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:35.307 INFO 4486 --- [ocp-async-1,1939a0e883178b3f,b49825d6fe0d53a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:37.063 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:50:37.064 INFO 4486 --- [ocp-schedules-12,09ec119c3686ab6e,c0ceb8238491c62f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:50:37.066 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:50:37.066 INFO 4486 --- [ocp-schedules-19,287065ebe82ee2a2,f91b6cb41ad1fc52] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:50:37.068 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:50:37.069 INFO 4486 --- [ocp-schedules-19,287065ebe82ee2a2,f91b6cb41ad1fc52] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:37.069 INFO 4486 --- [ocp-schedules-12,09ec119c3686ab6e,c0ceb8238491c62f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:50:37.081 INFO 4486 --- [ocp-schedules-7,8698152e28ed0f6a,5d8ec4400f2112b2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:50:37.083 INFO 4486 --- [ocp-schedules-7,8698152e28ed0f6a,5d8ec4400f2112b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:50:37.088 INFO 4486 --- [ocp-schedules-15,3aacc4b42137ab3f,992bb0af7aa178c7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:50:37.089 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:50:37.089 INFO 4486 --- [ocp-schedules-29,76fb8143ca884fa8,d4b0c02127af73be] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:50:37.090 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:50:37.091 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:50:37.091 INFO 4486 --- [ocp-schedules-27,5a7d12d60b5037a5,b4d81a033f5cb732] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:50:37.091 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:50:37.092 INFO 4486 --- [host-check-clockdiff1,76fb8143ca884fa8,db01561847d20c06] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:50:37.092 INFO 4486 --- [host-check-agent-status1,5a7d12d60b5037a5,9e0f954edd0fd979] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:50:37.093 INFO 4486 --- [host-check-agent-status2,5a7d12d60b5037a5,74e16ab30002bf7c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:50:37.093 INFO 4486 --- [host-check-agent-status3,5a7d12d60b5037a5,106c04d2f9b7f507] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:50:37.093 INFO 4486 --- [host-check-agent-status4,5a7d12d60b5037a5,77531edae1bbd2e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:50:37.094 INFO 4486 --- [host-check-clockdiff2,76fb8143ca884fa8,f7ce89abbc8f06b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:50:37.094 INFO 4486 --- [host-check-clockdiff3,76fb8143ca884fa8,3d1d3aef018c3940] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:50:37.096 INFO 4486 --- [host-check-clockdiff4,76fb8143ca884fa8,7e02ed0717284770] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:50:37.097 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:50:37.099 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:37.100 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:50:37.100 INFO 4486 --- [host-check-agent-status3,5a7d12d60b5037a5,106c04d2f9b7f507] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:50:37.101 INFO 4486 --- [host-check-agent-status4,5a7d12d60b5037a5,77531edae1bbd2e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:50:37.101 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:50:37.102 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:50:37.102 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:37.128 INFO 4486 --- [host-check-agent-status3,5a7d12d60b5037a5,106c04d2f9b7f507] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:50:37.146 INFO 4486 --- [host-check-agent-status2,5a7d12d60b5037a5,74e16ab30002bf7c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:50:37.128 INFO 4486 --- [host-check-agent-status1,5a7d12d60b5037a5,9e0f954edd0fd979] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:37.146 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:50:37.147 INFO 4486 --- [host-check-agent-status4,5a7d12d60b5037a5,77531edae1bbd2e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:50:37.150 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:50:37.150 INFO 4486 --- [host-check-agent-status2,5a7d12d60b5037a5,74e16ab30002bf7c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:50:37.150 INFO 4486 --- [host-check-agent-status1,5a7d12d60b5037a5,9e0f954edd0fd979] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:37.151 INFO 4486 --- [host-check-clockdiff2,76fb8143ca884fa8,f7ce89abbc8f06b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057837 4 4 , error= 2024-10-16 13:50:37.154 INFO 4486 --- [host-check-clockdiff4,76fb8143ca884fa8,7e02ed0717284770] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057837 4 4 , error= 2024-10-16 13:50:37.157 INFO 4486 --- [host-check-clockdiff3,76fb8143ca884fa8,3d1d3aef018c3940] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057837 5 5 , error= 2024-10-16 13:50:37.161 INFO 4486 --- [host-check-clockdiff1,76fb8143ca884fa8,db01561847d20c06] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057837 0 0 , error= 2024-10-16 13:50:37.167 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.168 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:37.170 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:50:37.179 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:50:37.182 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:50:37.183 INFO 4486 --- [host-check-agent-status4,5a7d12d60b5037a5,77531edae1bbd2e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:50:37.183 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:50:37.186817872+08:00 2024-10-16 13:50:37.184 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:50:37.183+08:00 2024-10-16 13:50:37.184 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:50:37.184296542+08:00 2024-10-16 13:50:37.185 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:50:37.184+08:00 2024-10-16 13:50:37.185 INFO 4486 --- [host-check-agent-status3,5a7d12d60b5037a5,106c04d2f9b7f507] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:50:37.186 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.186 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:50:37.187 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:37.189 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:50:37.192900037+08:00 2024-10-16 13:50:37.189 INFO 4486 --- [host-check-agent-status1,5a7d12d60b5037a5,9e0f954edd0fd979] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:50:37.189 INFO 4486 --- [host-check-agent-status4,5a7d12d60b5037a5,77531edae1bbd2e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:50:37.189 INFO 4486 --- [host-check-agent-status4,5a7d12d60b5037a5,77531edae1bbd2e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:50:37.189 INFO 4486 --- [host-check-agent-status2,5a7d12d60b5037a5,74e16ab30002bf7c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:50:37.189 INFO 4486 --- [host-check-agent-status4,5a7d12d60b5037a5,77531edae1bbd2e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:50:37.190 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:50:37.189+08:00 2024-10-16 13:50:37.191 INFO 4486 --- [host-check-agent-status3,5a7d12d60b5037a5,106c04d2f9b7f507] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:50:37.191 INFO 4486 --- [host-check-agent-status3,5a7d12d60b5037a5,106c04d2f9b7f507] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:50:37.191 INFO 4486 --- [host-check-agent-status3,5a7d12d60b5037a5,106c04d2f9b7f507] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:50:37.191 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:50:37.192 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:50:37.183+08:00 and status AVAILABLE 2024-10-16 13:50:37.193 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:50:37.189+08:00 and status AVAILABLE 2024-10-16 13:50:37.193 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:50:37.184+08:00 and status AVAILABLE 2024-10-16 13:50:37.193 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.193 INFO 4486 --- [host-check-agent-status2,5a7d12d60b5037a5,74e16ab30002bf7c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:50:37.193 INFO 4486 --- [host-check-agent-status2,5a7d12d60b5037a5,74e16ab30002bf7c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:50:37.193 INFO 4486 --- [host-check-agent-status2,5a7d12d60b5037a5,74e16ab30002bf7c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:50:37.194 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:50:37.195 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:50:37.198900985+08:00 2024-10-16 13:50:37.195 INFO 4486 --- [host-check-agent-status1,5a7d12d60b5037a5,9e0f954edd0fd979] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:50:37.195 INFO 4486 --- [host-check-agent-status1,5a7d12d60b5037a5,9e0f954edd0fd979] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:50:37.195 INFO 4486 --- [host-check-agent-status1,5a7d12d60b5037a5,9e0f954edd0fd979] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:50:37.196 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:50:37.196 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:50:37.196 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:50:37.195+08:00 2024-10-16 13:50:37.196 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:50:37.197 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:50:37.198 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.199 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:50:37.195+08:00 and status AVAILABLE 2024-10-16 13:50:37.201 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:50:37.202 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:50:37.203 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.204 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.205 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:50:37.205 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:50:37.206 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.206 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:37.206 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:50:37.207 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:50:37.207 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:37.208 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:37.208 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:50:37.211 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:50:37.212 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:50:37.213 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.213 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.213 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.213 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.213 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.214 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.218 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.219 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:37.230 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.232 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:37.232 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:37.233 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:37.234 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:50:37.235 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:50:37.237 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:50:37.238 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.240 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:50:37.254 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.256 INFO 4486 --- [ocp-schedules-6,647f8256b90222cc,657dca81f0bd6bc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:50:37.261 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:50:37.261 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:50:37.272 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:37.272 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:50:37.272 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:50:37.274 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:50:37.283 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:50:37.294 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:50:37.297 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:37.298 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:37.299 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:37.305 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:50:37.307 INFO 4486 --- [ocp-async-3,cb9205f186311faa,da53f6684742abe8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:37.326 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:50:37.327 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:50:37.328 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:50:37.334 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:50:37.337 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:50:37.339 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:50:37.345 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:50:37.354 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:37.355 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:37.363 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:37.383 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:50:37.386 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:50:37.389 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:50:37.395 INFO 4486 --- [host-check-host-status3,3aacc4b42137ab3f,7e9b0ecae94de69a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:50:37.396 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:50:37.402 INFO 4486 --- [host-check-host-status2,3aacc4b42137ab3f,ca3f6115ef99d85b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:50:37.408 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:50:37.413 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:50:37.423 INFO 4486 --- [host-check-host-status4,3aacc4b42137ab3f,742e9ef007e659a5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:50:37.467 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:37.469 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.470 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:50:37.471 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.472 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.473 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:50:37.475 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:50:37.483 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.483 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.483 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.483 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.483 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.483 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.489 INFO 4486 --- [host-check-host-status1,3aacc4b42137ab3f,fca1f8ee762a2767] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:50:37.513 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:50:37.555 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:50:37.608 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:37.623 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:37.628 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.629 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:50:37.630 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:50:37.635 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.636 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:50:37.637 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.638 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.639 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:50:37.647 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.647 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.647 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.647 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.647 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.650 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.657 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:50:37.902 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a8198bde05980ecd,e5e9cad2eb374fee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a8198bde05980ecd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:37.903 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a8198bde05980ecd,e5e9cad2eb374fee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a8198bde05980ecd, duration=3 ms] 2024-10-16 13:50:37.907 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:50:37.956 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:37.958 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.959 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:50:37.960 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.960 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.961 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:50:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:50:37.974 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.974 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.974 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.974 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.974 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:37.975 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:50:37.994 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:50:37.994 INFO 4486 --- [prometheus-minute-schedule-1,5a5da5b1ad63e75f,aefbaefafe6db49f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:38.000 INFO 4486 --- [prometheus-minute-schedule-1,5a5da5b1ad63e75f,aefbaefafe6db49f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:38.185 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:50:38.235 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:38.238 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:38.242 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:50:38.243 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:38.243 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:38.244 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:50:38.253 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:38.253 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:38.253 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:38.253 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:38.253 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:50:38.254 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:38.315 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:50:38.449 INFO 4486 --- [ocp-schedules-14,b26406c7773ab1ec,24dd6f45ca4644d1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:50:38.449 INFO 4486 --- [ocp-schedules-14,b26406c7773ab1ec,24dd6f45ca4644d1] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:50:38.450 INFO 4486 --- [ocp-schedules-14,b26406c7773ab1ec,24dd6f45ca4644d1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:50:38.526 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:50:38.319772, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:50:38.526 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:50:38.319772, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:50:38.568 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:50:38.319772} } 2024-10-16 13:50:38.630 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:50:38.639 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:50:38.641 INFO 4486 --- [ocp-schedules-3,2516a03fa5cce980,4a55bf895073d21c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:50:38.852 INFO 4486 --- [ocp-schedules-9,0137b3ffab4d19b0,d555e013a7018b80] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:50:38.853 INFO 4486 --- [ocp-schedules-9,0137b3ffab4d19b0,d555e013a7018b80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:39.110 INFO 4486 --- [ocp-schedules-2,b97d2e7cfcb7f481,d81b25f8a81cb904] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:50:39.307 INFO 4486 --- [ocp-async-4,97c3d5c0a1182198,b069baab3a838a08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:39.425 INFO 4486 --- [ocp-schedules-23,913816dc573b4b9c,ee9c85ec0f3eace0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:50:39.462 WARN 4486 --- [scanner-13293022-1,913816dc573b4b9c,9c1d5a91a6c5780a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:50:40.007 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:50:40.008 INFO 4486 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:50:40.012 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,080211abc290ee78,ca19f5af80c4c2d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=080211abc290ee78, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:40.012 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,080211abc290ee78,ca19f5af80c4c2d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=080211abc290ee78, duration=1 ms] 2024-10-16 13:50:40.029 INFO 4486 --- [Check obproxy1,bed1026f261303d8,74e093352a3b1968] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:50:40.030 INFO 4486 --- [Check obproxy2,f59098d5d4a8975d,81c1259cd467aa2e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:50:40.030 INFO 4486 --- [Collect obproxy2,f594aa1eace7490b,f8329e70ed0da5c7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:50:40.032 INFO 4486 --- [Collect obproxy1,f559bddb18e6787f,771f0e741227f9d6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:50:40.034 INFO 4486 --- [Check obproxy1,bed1026f261303d8,74e093352a3b1968] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:40.038 INFO 4486 --- [Check obproxy2,f59098d5d4a8975d,81c1259cd467aa2e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:40.039 INFO 4486 --- [Collect obproxy2,f594aa1eace7490b,f8329e70ed0da5c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:40.041 INFO 4486 --- [Collect obproxy1,f559bddb18e6787f,771f0e741227f9d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:50:40.061 INFO 4486 --- [Check obproxy2,f59098d5d4a8975d,81c1259cd467aa2e] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:50:40.061 INFO 4486 --- [Check obproxy2,f59098d5d4a8975d,81c1259cd467aa2e] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:50:40.062 INFO 4486 --- [Collect obproxy2,f594aa1eace7490b,f8329e70ed0da5c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:50:40.066 INFO 4486 --- [Check obproxy1,bed1026f261303d8,74e093352a3b1968] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:50:40.066 INFO 4486 --- [Check obproxy1,bed1026f261303d8,74e093352a3b1968] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:50:40.068 INFO 4486 --- [Collect obproxy2,f594aa1eace7490b,f8329e70ed0da5c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:50:40.072 INFO 4486 --- [Collect obproxy2,f594aa1eace7490b,f8329e70ed0da5c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:50:40.074 INFO 4486 --- [Collect obproxy1,f559bddb18e6787f,771f0e741227f9d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:50:40.080 INFO 4486 --- [Collect obproxy1,f559bddb18e6787f,771f0e741227f9d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:50:40.085 INFO 4486 --- [Collect obproxy1,f559bddb18e6787f,771f0e741227f9d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:50:40.273 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:40.906 INFO 4486 --- [ocp-schedules-1,37f3baedb6777840,bdb223de323f2e18] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:50:40.994 INFO 4486 --- [prometheus-minute-schedule-1,a7ac048c3fba5313,457d6fd942cd4670] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:40.995 INFO 4486 --- [prometheus-hour-schedule-1,b9a7ca0b35856ae2,e52ecd0f5d4772ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:50:41.000 INFO 4486 --- [prometheus-minute-schedule-1,a7ac048c3fba5313,457d6fd942cd4670] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:41.307 INFO 4486 --- [ocp-async-10,1b666a6af4793b3c,0d72d6f5740af805] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:41.547 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,86e93ae5f7aaaed0,111356b2e5059263] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=86e93ae5f7aaaed0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:41.547 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,86e93ae5f7aaaed0,111356b2e5059263] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=86e93ae5f7aaaed0, duration=1 ms] 2024-10-16 13:50:41.856 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,996565357ecb34ae,a599666c04d7a96b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=996565357ecb34ae, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:41.856 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,996565357ecb34ae,a599666c04d7a96b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=996565357ecb34ae, duration=1 ms] 2024-10-16 13:50:42.141 INFO 4486 --- [ocp-schedules-4,c806b79304d04bac,0a670d81e1ca3f65] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:50:42.540 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e9271a6bdcce4903,26bbb108c1bc16fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=e9271a6bdcce4903, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:50:42.541 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e9271a6bdcce4903,26bbb108c1bc16fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=e9271a6bdcce4903, duration=2 ms] 2024-10-16 13:50:43.012 INFO 4486 --- [ocp-async-1,ef2a622f19d22e0c,396ccb039261cbfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:50:43.274 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:43.307 INFO 4486 --- [ocp-async-5,699f71c6a536f269,73f70798df994acd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:43.539 INFO 4486 --- [ocp-schedules-24,c9c55ee931fc2612,6e88f022dfd30f5b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:50:43.541 INFO 4486 --- [ocp-schedules-24,c9c55ee931fc2612,6e88f022dfd30f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:43.542 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:50:43.542 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:50:43.542 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:50:43.994 INFO 4486 --- [prometheus-minute-schedule-1,00d42d740e696f05,8f13436d9cbb9c61] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:44.000 INFO 4486 --- [prometheus-minute-schedule-1,00d42d740e696f05,8f13436d9cbb9c61] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:45.308 INFO 4486 --- [ocp-async-4,c33fbfb3cc470975,d9e60e28053390ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:45.535 INFO 4486 --- [ocp-schedules-11,ab0524595b495c23,2276a21ec5d82021] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:50:45.540 INFO 4486 --- [ocp-schedules-11,ab0524595b495c23,2276a21ec5d82021] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:50:45.540 INFO 4486 --- [ocp-schedules-11,ab0524595b495c23,2276a21ec5d82021] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:50:45.564 INFO 4486 --- [ocp-schedules-5,d5f996f8450b1eaf,95906bf883b0bca1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:50:45.564 INFO 4486 --- [ocp-schedules-5,d5f996f8450b1eaf,95906bf883b0bca1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:50:46.275 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:46.994 INFO 4486 --- [prometheus-minute-schedule-1,f58472537391f1e3,cc190e7d7f9a194a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:47.000 INFO 4486 --- [prometheus-minute-schedule-1,f58472537391f1e3,cc190e7d7f9a194a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:47.071 INFO 4486 --- [ocp-schedules-19,287065ebe82ee2a2,f91b6cb41ad1fc52] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:50:47.192 INFO 4486 --- [ocp-schedules-7,8698152e28ed0f6a,5d8ec4400f2112b2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:50:47.201 INFO 4486 --- [ocp-schedules-5,a585f06050a0a9a2,37993a706edce747] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:50:47.308 INFO 4486 --- [ocp-async-10,9a965e4e78388380,279dd45b705b8ed8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:47.908 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,508a3b5e52e347a8,4582b3977020e3ec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=508a3b5e52e347a8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:47.908 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,508a3b5e52e347a8,4582b3977020e3ec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=508a3b5e52e347a8, duration=1 ms] 2024-10-16 13:50:48.879 INFO 4486 --- [ocp-schedules-9,0137b3ffab4d19b0,d555e013a7018b80] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:50:49.276 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:49.308 INFO 4486 --- [ocp-async-8,ad2a1569aecc4a81,fed9ab0bc7ee3b8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:49.570 INFO 4486 --- [ocp-schedules-23,5d53c18235146003,9e2b9a0a1e6e6a11] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:50:49.617 WARN 4486 --- [scanner-13293296-1,5d53c18235146003,f0a8fafb490e18da] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:50:49.693 INFO 4486 --- [alarm-detect-14,5d53c18235146003,c7b6d58b3464a8c8] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057815 ~ 1729057839 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057830, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057830, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057830=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:50:49.699 INFO 4486 --- [alarm-detect-14,5d53c18235146003,c7b6d58b3464a8c8] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:50:49.994 INFO 4486 --- [prometheus-minute-schedule-1,76adb8cda653ab30,b0de581cf1efefd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:50.000 INFO 4486 --- [prometheus-minute-schedule-1,76adb8cda653ab30,b0de581cf1efefd0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:50.019 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8aba3bbb0e731f9b,6a5f2f871c708c0b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8aba3bbb0e731f9b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:50.019 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8aba3bbb0e731f9b,6a5f2f871c708c0b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8aba3bbb0e731f9b, duration=1 ms] 2024-10-16 13:50:50.486 INFO 4486 --- [ocp-schedules-13,c9ae2fa3ee2b5607,5a59ceb06df35c87] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:50:50.486 INFO 4486 --- [ocp-schedules-13,c9ae2fa3ee2b5607,5a59ceb06df35c87] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:50:50.492 INFO 4486 --- [ocp-schedules-13,c9ae2fa3ee2b5607,5a59ceb06df35c87] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:50:50.492 INFO 4486 --- [ocp-schedules-13,c9ae2fa3ee2b5607,5a59ceb06df35c87] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:50:50.496 INFO 4486 --- [ocp-schedules-13,c9ae2fa3ee2b5607,5a59ceb06df35c87] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:50:50.496 INFO 4486 --- [ocp-schedules-13,c9ae2fa3ee2b5607,5a59ceb06df35c87] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:50:50.912 INFO 4486 --- [ocp-schedules-25,8200864704d755e0,1b94a09be4e34eaa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:50:50.912 INFO 4486 --- [ocp-schedules-27,49e377e6a0943d04,3c529eb098b7a49d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:50:50.912 INFO 4486 --- [ocp-schedules-27,49e377e6a0943d04,3c529eb098b7a49d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:50:50.912 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:50:50.912 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:50:50.912 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:50:50.912 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:50:50.912 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:50:50.918 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:50.918 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:50:50.918 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:50:50.918 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:50:50.918 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:50:50.919 INFO 4486 --- [ocp-schedules-27,49e377e6a0943d04,3c529eb098b7a49d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:50:50.920 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:50.920 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:50:50.920 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:50:50.920 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:50:50.920 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:50:50.920 INFO 4486 --- [ocp-schedules-27,49e377e6a0943d04,3c529eb098b7a49d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:50:50.945 INFO 4486 --- [ocp-schedules-27,49e377e6a0943d04,3c529eb098b7a49d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:50:50.945 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:50:50.946 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:50:50.946 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:50:50.947 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:50:50.949 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:50:50.965 INFO 4486 --- [ocp-schedules-27,49e377e6a0943d04,3c529eb098b7a49d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:50:51.308 INFO 4486 --- [ocp-async-7,d54fb0bde08ba51e,22633c0df3147203] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:51.552 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,73c44dd8305bf363,4fac26e06b659d18] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=73c44dd8305bf363, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:51.553 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,73c44dd8305bf363,4fac26e06b659d18] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=73c44dd8305bf363, duration=2 ms] 2024-10-16 13:50:51.860 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5501f2d38c96676e,7f84ad9841b48339] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5501f2d38c96676e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:51.861 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5501f2d38c96676e,7f84ad9841b48339] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5501f2d38c96676e, duration=1 ms] 2024-10-16 13:50:52.278 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:52.994 INFO 4486 --- [prometheus-minute-schedule-1,d3977182de1eb6a4,ca5fd0534bccbd05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:53.000 INFO 4486 --- [prometheus-minute-schedule-1,d3977182de1eb6a4,ca5fd0534bccbd05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:53.012 INFO 4486 --- [ocp-async-6,3c6fbcc0cd208b12,b3c4dd9cfe52ecff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:50:53.308 INFO 4486 --- [ocp-async-10,4b2f569bc0f5251a,1f5b8373a95f7c85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:53.542 INFO 4486 --- [ocp-schedules-24,c9c55ee931fc2612,6e88f022dfd30f5b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:50:54.625 INFO 4486 --- [ocp-schedules-16,1ece14fd881c8303,94d47b92ba66d740] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:50:54.625 INFO 4486 --- [ocp-schedules-16,e2c569d170d44379,9d9741e01e049547] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:50:54.627 INFO 4486 --- [ocp-schedules-16,e2c569d170d44379,9d9741e01e049547] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:50:54.657 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.658 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:54.669 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.671 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:54.693 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:54.693 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.694 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:54.694 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.695 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:50:54.725 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:54.726 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.727 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:54.728 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:50:54.728 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.731 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:50:54.732 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:54.742 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:54.742 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:54.754 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:54.763 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:54.764 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.765 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:54.766 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:50:54.771 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:50:54.776 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:50:54.778 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:50:54.792 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:54.794 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.795 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:50:54.799 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:50:54.834 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:50:54.834 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:50:54.844 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:54.845 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.846 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:50:54.849 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.853 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:50:54.856 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:54.879 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:50:54.888 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:54.888 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:54.890 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:54.891 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:54.892 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:50:54.902 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:50:54.902 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:50:54.917 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:54.917 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:54.945 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:54.945 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:54.974 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:54.974 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:50:54.999 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:54.999 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:50:55.024 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:50:55.050 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:55.051 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:55.052 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:50:55.056 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:50:55.061 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:50:55.065 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:50:55.066 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:50:55.090 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:50:55.229 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:50:55.255 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:50:55.257 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:50:55.258 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:50:55.279 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:55.286 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:50:55.286 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:50:55.309 INFO 4486 --- [ocp-async-5,eb3594769f69ee56,052d8b8f036b7f2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:55.994 INFO 4486 --- [prometheus-minute-schedule-1,63e4e685efb7fb3c,542dc26206ee354f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:56.000 INFO 4486 --- [prometheus-minute-schedule-1,63e4e685efb7fb3c,542dc26206ee354f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:57.309 INFO 4486 --- [ocp-async-6,400ef7efa4cd3c3b,69e0da17b62db8fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:57.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a5d4dd061c05cd10,1ec22155bb644f31] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a5d4dd061c05cd10, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:50:57.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a5d4dd061c05cd10,1ec22155bb644f31] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a5d4dd061c05cd10, duration=1 ms] 2024-10-16 13:50:58.282 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:50:58.994 INFO 4486 --- [prometheus-minute-schedule-1,f22168896c65fa21,43b34bae1a158893] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057860, interval=60 2024-10-16 13:50:59.000 INFO 4486 --- [prometheus-minute-schedule-1,f22168896c65fa21,43b34bae1a158893] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:50:59.114 INFO 4486 --- [ocp-schedules-29,8b4b61e0a73a8722,98dd4623fa8761b7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:50:59.309 INFO 4486 --- [ocp-async-1,dcd19b358ed0057f,fce8ee73a4fd925c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:50:59.781 INFO 4486 --- [ocp-schedules-1,72b21196d081e41b,13d557476281b101] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:50:59.820 WARN 4486 --- [scanner-13293651-1,72b21196d081e41b,ad91dbc973360956] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:50:59.854 INFO 4486 --- [alarm-detect-8,72b21196d081e41b,96615e985b3e37e1] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057830 ~ 1729057849 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057845, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057845, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057845=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:51:00.029 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,715bc1e167cdaa9f,d183cab8b9afd8f7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=715bc1e167cdaa9f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:00.029 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,715bc1e167cdaa9f,d183cab8b9afd8f7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=715bc1e167cdaa9f, duration=1 ms] 2024-10-16 13:51:00.917 INFO 4486 --- [ocp-schedules-23,690036a3f4c6c5a1,6f653d257d96a023] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:51:01.283 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:01.309 INFO 4486 --- [ocp-async-10,1af28201c539067c,582dfddb82823c85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:01.557 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3c0dde0cf997700c,18badaa27fa4d35a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3c0dde0cf997700c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:01.557 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3c0dde0cf997700c,18badaa27fa4d35a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3c0dde0cf997700c, duration=1 ms] 2024-10-16 13:51:01.866 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d348a073b95967f8,5a7767807864a26f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d348a073b95967f8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:01.866 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d348a073b95967f8,5a7767807864a26f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d348a073b95967f8, duration=2 ms] 2024-10-16 13:51:01.994 INFO 4486 --- [prometheus-minute-schedule-1,7e5148219426dbf9,4a53a7ebf9ca1854] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:02.000 INFO 4486 --- [prometheus-minute-schedule-1,7e5148219426dbf9,4a53a7ebf9ca1854] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:02.098 INFO 4486 --- [ocp-schedules-29,6b64234ff2cdfc9c,5b4aa16d8c5507e4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:51:02.098+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:51:02.101 INFO 4486 --- [ocp-schedules-29,6b64234ff2cdfc9c,5b4aa16d8c5507e4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:51:02.105 INFO 4486 --- [ocp-schedules-29,6b64234ff2cdfc9c,5b4aa16d8c5507e4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:51:02.116 INFO 4486 --- [ocp-schedules-29,6b64234ff2cdfc9c,5b4aa16d8c5507e4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:51:03.012 INFO 4486 --- [ocp-async-3,53efd16cbc6e1287,8e7bff5e51c45619] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:51:03.310 INFO 4486 --- [ocp-async-4,05b2dbf6021b8b2d,18e29060dfed9aa7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:04.284 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:04.994 INFO 4486 --- [prometheus-minute-schedule-1,07727782ac565ef2,10211df4be00ffe4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:05.001 INFO 4486 --- [prometheus-minute-schedule-1,07727782ac565ef2,10211df4be00ffe4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:05.310 INFO 4486 --- [ocp-async-2,85862ab7269b74ba,4de467230834c1c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:07.066 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:51:07.066 INFO 4486 --- [ocp-schedules-27,5516d7fc65f236e7,b75bdcf5920c1f70] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:51:07.068 INFO 4486 --- [ocp-schedules-27,5516d7fc65f236e7,b75bdcf5920c1f70] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:07.081 INFO 4486 --- [ocp-schedules-2,da2cea6af884e571,9d5b413c1ce50ee1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:51:07.083 INFO 4486 --- [ocp-schedules-2,da2cea6af884e571,9d5b413c1ce50ee1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:51:07.088 INFO 4486 --- [ocp-schedules-14,1bd9e6c9a5e8e90e,bab94d48b3a46bdc] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:51:07.089 INFO 4486 --- [ocp-schedules-17,961e273d915eb248,237d1783abd0c623] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:51:07.090 INFO 4486 --- [ocp-schedules-3,99e7029f8805eb17,453c6e07e90327a2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:51:07.090 INFO 4486 --- [host-check-clockdiff2,1bd9e6c9a5e8e90e,880278a875eea1d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:51:07.090 INFO 4486 --- [host-check-clockdiff1,1bd9e6c9a5e8e90e,527ed0fbdb1fe613] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:51:07.090 INFO 4486 --- [host-check-clockdiff3,1bd9e6c9a5e8e90e,8e538c61653737a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:51:07.090 INFO 4486 --- [host-check-clockdiff4,1bd9e6c9a5e8e90e,cdf6a37ca227c728] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:51:07.091 INFO 4486 --- [host-check-agent-status1,961e273d915eb248,1acefed9364c688a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:51:07.091 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:51:07.092 INFO 4486 --- [host-check-agent-status2,961e273d915eb248,b7002083e9dd8352] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:51:07.093 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:51:07.094 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:51:07.094 INFO 4486 --- [host-check-agent-status4,961e273d915eb248,6e73302ba4fdbf03] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:51:07.096 INFO 4486 --- [host-check-agent-status3,961e273d915eb248,fa69e33f39bfb471] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:51:07.096 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:51:07.110 INFO 4486 --- [host-check-agent-status1,961e273d915eb248,1acefed9364c688a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:07.110 INFO 4486 --- [host-check-agent-status2,961e273d915eb248,b7002083e9dd8352] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:51:07.107 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:07.133 INFO 4486 --- [host-check-clockdiff3,1bd9e6c9a5e8e90e,8e538c61653737a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057867 4 4 , error= 2024-10-16 13:51:07.138 INFO 4486 --- [host-check-clockdiff1,1bd9e6c9a5e8e90e,527ed0fbdb1fe613] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057867 0 0 , error= 2024-10-16 13:51:07.129 INFO 4486 --- [host-check-agent-status4,961e273d915eb248,6e73302ba4fdbf03] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:51:07.139 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:07.139 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:51:07.139 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:51:07.140 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:51:07.140 INFO 4486 --- [host-check-agent-status1,961e273d915eb248,1acefed9364c688a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:07.140 INFO 4486 --- [host-check-agent-status2,961e273d915eb248,b7002083e9dd8352] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:51:07.140 INFO 4486 --- [host-check-agent-status4,961e273d915eb248,6e73302ba4fdbf03] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:51:07.140 INFO 4486 --- [host-check-agent-status3,961e273d915eb248,fa69e33f39bfb471] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:51:07.141 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:51:07.141 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:51:07.141 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:51:07.142 INFO 4486 --- [host-check-agent-status3,961e273d915eb248,fa69e33f39bfb471] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:51:07.145 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.146 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:07.147 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:51:07.167 INFO 4486 --- [host-check-agent-status4,961e273d915eb248,6e73302ba4fdbf03] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:51:07.167 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:51:07.168 INFO 4486 --- [host-check-agent-status1,961e273d915eb248,1acefed9364c688a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:51:07.168 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:51:07.168 INFO 4486 --- [host-check-agent-status2,961e273d915eb248,b7002083e9dd8352] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:51:07.169 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:51:07.170 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:51:07.170072310+08:00 2024-10-16 13:51:07.170 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:51:07.173600040+08:00 2024-10-16 13:51:07.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.171 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:51:07.170+08:00 2024-10-16 13:51:07.171 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:51:07.170+08:00 2024-10-16 13:51:07.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:07.172 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:51:07.176083296+08:00 2024-10-16 13:51:07.172 INFO 4486 --- [host-check-agent-status3,961e273d915eb248,fa69e33f39bfb471] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:51:07.172 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:51:07.172+08:00 2024-10-16 13:51:07.173 INFO 4486 --- [host-check-agent-status4,961e273d915eb248,6e73302ba4fdbf03] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:51:07.173 INFO 4486 --- [host-check-agent-status4,961e273d915eb248,6e73302ba4fdbf03] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:51:07.173 INFO 4486 --- [host-check-agent-status4,961e273d915eb248,6e73302ba4fdbf03] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:51:07.173 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:51:07.173 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:51:07.170+08:00 and status AVAILABLE 2024-10-16 13:51:07.173 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:51:07.170+08:00 and status AVAILABLE 2024-10-16 13:51:07.174 INFO 4486 --- [host-check-agent-status2,961e273d915eb248,b7002083e9dd8352] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:51:07.174 INFO 4486 --- [host-check-agent-status2,961e273d915eb248,b7002083e9dd8352] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:51:07.174 INFO 4486 --- [host-check-agent-status2,961e273d915eb248,b7002083e9dd8352] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:51:07.174 INFO 4486 --- [host-check-agent-status1,961e273d915eb248,1acefed9364c688a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:51:07.174 INFO 4486 --- [host-check-agent-status1,961e273d915eb248,1acefed9364c688a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:51:07.174 INFO 4486 --- [host-check-agent-status1,961e273d915eb248,1acefed9364c688a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:51:07.174 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:51:07.172+08:00 and status AVAILABLE 2024-10-16 13:51:07.175 INFO 4486 --- [host-check-agent-status3,961e273d915eb248,fa69e33f39bfb471] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:51:07.176 INFO 4486 --- [host-check-agent-status3,961e273d915eb248,fa69e33f39bfb471] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:51:07.176 INFO 4486 --- [host-check-agent-status3,961e273d915eb248,fa69e33f39bfb471] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:51:07.176 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:51:07.176 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:51:07.180111151+08:00 2024-10-16 13:51:07.176 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:51:07.176 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:51:07.176 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:51:07.176 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:51:07.176+08:00 2024-10-16 13:51:07.178 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.179 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:51:07.176+08:00 and status AVAILABLE 2024-10-16 13:51:07.182 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:51:07.182 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:51:07.185 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:07.185 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:51:07.185 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:51:07.187 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:51:07.187 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:07.187 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:51:07.191 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.191 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:51:07.192 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:51:07.192 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:51:07.201 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.202 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:51:07.208 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:07.209 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:07.211 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.211 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:07.215 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:51:07.216 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:07.217 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.218 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:51:07.227 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:07.228 INFO 4486 --- [ocp-schedules-1,81207800f7fbae0e,e9bf92c03d6cc03c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:51:07.237 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:51:07.242 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:51:07.254 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:51:07.255 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:51:07.256 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:51:07.268 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:51:07.269 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:07.270 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:07.283 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:51:07.286 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:07.288 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:07.294 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:51:07.303 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:51:07.306 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:51:07.310 INFO 4486 --- [ocp-async-10,cff2e6f8fc573351,2fca795dacd1c489] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:07.310 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:07.313 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:51:07.315 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:51:07.326 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:07.329 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:51:07.331 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:07.340 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:51:07.354 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:51:07.355 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:07.361 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:51:07.364 INFO 4486 --- [host-check-host-status3,99e7029f8805eb17,786ac13bdf418783] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:51:07.364 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:51:07.372 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:51:07.374 INFO 4486 --- [host-check-host-status4,99e7029f8805eb17,bedf41a68de65bc2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:51:07.386 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:07.393 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:51:07.404 INFO 4486 --- [host-check-host-status2,99e7029f8805eb17,83005a55229c3128] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:51:07.445 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:51:07.456 INFO 4486 --- [host-check-host-status1,99e7029f8805eb17,47d2aca62409d9db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:51:07.944 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c15b88fb8cc9f87c,2a3be47cc78e7cd5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c15b88fb8cc9f87c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:07.944 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c15b88fb8cc9f87c,2a3be47cc78e7cd5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c15b88fb8cc9f87c, duration=2 ms] 2024-10-16 13:51:07.994 INFO 4486 --- [prometheus-minute-schedule-1,30f76f8696aa07d0,df9bdca903f0cf0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:08.002 INFO 4486 --- [prometheus-minute-schedule-1,30f76f8696aa07d0,df9bdca903f0cf0f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:08.057 INFO 4486 --- [host-check-clockdiff4,1bd9e6c9a5e8e90e,cdf6a37ca227c728] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057868 4 4 , error= 2024-10-16 13:51:09.310 INFO 4486 --- [ocp-async-6,b59f680070bb35a6,ebbdde821bdc971f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:09.910 INFO 4486 --- [ocp-schedules-18,1cbd0821c15370d4,d28f6e1f33ff52f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:51:09.944 WARN 4486 --- [scanner-13293854-1,1cbd0821c15370d4,44daf573e8bcbe82] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:51:10.010 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:51:10.010 INFO 4486 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:51:10.040 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,35e09f339f8e3c54,05476b1317e61696] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=35e09f339f8e3c54, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:10.040 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,35e09f339f8e3c54,05476b1317e61696] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=35e09f339f8e3c54, duration=6 ms] 2024-10-16 13:51:10.044 INFO 4486 --- [Check obproxy2,905008f95bd63494,5a80563062706a1f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:51:10.045 INFO 4486 --- [Check obproxy2,905008f95bd63494,5a80563062706a1f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:10.045 INFO 4486 --- [Check obproxy1,f3bf798c21e712c2,c88eed7550308f91] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:51:10.045 INFO 4486 --- [Check obproxy1,f3bf798c21e712c2,c88eed7550308f91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:10.056 INFO 4486 --- [Collect obproxy1,d1982330928b64d8,e482d72663be7444] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:51:10.056 INFO 4486 --- [Collect obproxy1,d1982330928b64d8,e482d72663be7444] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:10.063 INFO 4486 --- [Collect obproxy2,fa37b6e83db5eb02,07a98504d36bf90b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:51:10.063 INFO 4486 --- [Collect obproxy2,fa37b6e83db5eb02,07a98504d36bf90b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:10.084 INFO 4486 --- [Check obproxy2,905008f95bd63494,5a80563062706a1f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:51:10.084 INFO 4486 --- [Check obproxy2,905008f95bd63494,5a80563062706a1f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:51:10.135 INFO 4486 --- [Collect obproxy1,d1982330928b64d8,e482d72663be7444] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:51:10.135 INFO 4486 --- [Collect obproxy2,fa37b6e83db5eb02,07a98504d36bf90b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:51:10.136 INFO 4486 --- [Check obproxy1,f3bf798c21e712c2,c88eed7550308f91] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:51:10.136 INFO 4486 --- [Check obproxy1,f3bf798c21e712c2,c88eed7550308f91] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:51:10.142 INFO 4486 --- [Collect obproxy1,d1982330928b64d8,e482d72663be7444] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:51:10.142 INFO 4486 --- [Collect obproxy2,fa37b6e83db5eb02,07a98504d36bf90b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:51:10.148 INFO 4486 --- [Collect obproxy2,fa37b6e83db5eb02,07a98504d36bf90b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:51:10.148 INFO 4486 --- [Collect obproxy1,d1982330928b64d8,e482d72663be7444] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:51:10.287 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:10.796 INFO 4486 --- [host-check-clockdiff2,1bd9e6c9a5e8e90e,880278a875eea1d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057870 4 4 , error= 2024-10-16 13:51:10.922 INFO 4486 --- [ocp-schedules-13,ef03b013ad95fc70,5fa99331d13c4738] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:51:10.977 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:51:10.994 INFO 4486 --- [prometheus-minute-schedule-1,8ec5aa2de7a6c594,ecfda5a2faf1e800] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:11.002 INFO 4486 --- [prometheus-minute-schedule-1,8ec5aa2de7a6c594,ecfda5a2faf1e800] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:51:11.310 INFO 4486 --- [ocp-async-1,41a6ef4c029139fa,f39d12d9e5a83e6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:11.560 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,65d860ad03759c2d,2238354c894b8d2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=65d860ad03759c2d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:11.561 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,65d860ad03759c2d,2238354c894b8d2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=65d860ad03759c2d, duration=1 ms] 2024-10-16 13:51:11.869 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2ec923fade6b7429,010db5c18f5fdf0c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2ec923fade6b7429, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:11.869 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2ec923fade6b7429,010db5c18f5fdf0c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2ec923fade6b7429, duration=0 ms] 2024-10-16 13:51:13.012 INFO 4486 --- [ocp-async-7,9877d4ee69adc44a,df2dd0f4e0c7c7aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:51:13.288 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:13.310 INFO 4486 --- [ocp-async-3,3c6eca5fb1603a17,2c334ed4a2e7d93f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:13.994 INFO 4486 --- [prometheus-minute-schedule-1,a44e2a90877c351b,cc119708f83e848a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:14.000 INFO 4486 --- [prometheus-minute-schedule-1,a44e2a90877c351b,cc119708f83e848a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:14.144 ERROR 4486 --- [metric-archive-second-25,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2323956) Duplicate entry '587073990944165888-1729057800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:51:14.142599] [YB420AC531B8-00060B7DC0C86E6C-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '587073990944165888-1729057800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:51:14.142599] [YB420AC531B8-00060B7DC0C86E6C-0-0] 2024-10-16 13:51:14.148 ERROR 4486 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '586774769061924864-1729057800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:51:14.148170] [YB420AC531B8-00060B7DB03B2A80-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '586774769061924864-1729057800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:51:14.148170] [YB420AC531B8-00060B7DB03B2A80-0-0] 2024-10-16 13:51:14.174 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057800, nodeCount=1000, affectRows=1000 2024-10-16 13:51:14.176 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057800, nodeCount=1000, affectRows=1000 2024-10-16 13:51:14.178 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057800, nodeCount=998, validCount=998 2024-10-16 13:51:14.179 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057800, nodeCount=1000, validCount=1000 2024-10-16 13:51:14.443 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057800, nodeCount=517, affectRows=517 2024-10-16 13:51:14.446 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057800, nodeCount=516, validCount=516 2024-10-16 13:51:14.646 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057800, nodeCount=933, affectRows=933 2024-10-16 13:51:14.649 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057800, nodeCount=933, validCount=933 2024-10-16 13:51:15.000 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057800, nodeCount=480, affectRows=480 2024-10-16 13:51:15.311 INFO 4486 --- [ocp-async-8,6dedb592458ff92d,3806e8cb26cd25c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:15.650 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057800, nodeCount=480, validCount=480 2024-10-16 13:51:16.059 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,592dc8f881736272,934ece8327a4bb59] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=592dc8f881736272, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:51:16.060 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,592dc8f881736272,934ece8327a4bb59] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=592dc8f881736272, duration=2 ms] 2024-10-16 13:51:16.289 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:16.994 INFO 4486 --- [prometheus-minute-schedule-1,77379307eb974482,8af10b3d0467bbf2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:17.001 INFO 4486 --- [prometheus-minute-schedule-1,77379307eb974482,8af10b3d0467bbf2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:17.070 INFO 4486 --- [ocp-schedules-27,5516d7fc65f236e7,b75bdcf5920c1f70] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:51:17.177 INFO 4486 --- [ocp-schedules-2,da2cea6af884e571,9d5b413c1ce50ee1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:51:17.204 INFO 4486 --- [ocp-schedules-4,5b74204e7ef664a9,22c3fa1fa206669e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:51:17.311 INFO 4486 --- [ocp-async-7,1821169407b73234,23928beb921e7a77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:17.950 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7e6ffdc1d57cd72b,0800aa7b6d3f7a55] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7e6ffdc1d57cd72b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:17.950 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7e6ffdc1d57cd72b,0800aa7b6d3f7a55] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7e6ffdc1d57cd72b, duration=1 ms] 2024-10-16 13:51:19.046 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5e4bc0c68726450d,adaf16d782849e3a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=5e4bc0c68726450d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:51:19.047 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5e4bc0c68726450d,adaf16d782849e3a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=5e4bc0c68726450d, duration=2 ms] 2024-10-16 13:51:19.119 INFO 4486 --- [ocp-schedules-27,5467495e01f7226f,91242c4ee2316285] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:51:19.290 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:19.311 INFO 4486 --- [ocp-async-9,e21cb0da876455d0,a4edeb10a41372ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:19.994 INFO 4486 --- [prometheus-minute-schedule-1,212dc25c3317b7c2,064d39ba6082a883] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:20.001 INFO 4486 --- [prometheus-minute-schedule-1,212dc25c3317b7c2,064d39ba6082a883] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:20.045 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ed471e45b7261b1a,8decff77ad1c6fc1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ed471e45b7261b1a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:20.045 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ed471e45b7261b1a,8decff77ad1c6fc1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ed471e45b7261b1a, duration=1 ms] 2024-10-16 13:51:20.160 INFO 4486 --- [ocp-schedules-30,10b8bedf57bdd2d7,34f0ec261beb9779] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:51:20.187 WARN 4486 --- [scanner-13294974-1,10b8bedf57bdd2d7,1235064dc613b7f9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:51:20.229 INFO 4486 --- [alarm-detect-1,10b8bedf57bdd2d7,b97b0090daf6e729] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057845 ~ 1729057870 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057860, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057860, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057860=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:51:20.928 INFO 4486 --- [ocp-schedules-11,ed00b7f47e7ebf49,f7de4ceb44384d67] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:51:20.996 INFO 4486 --- [prometheus-cache-monitor1,105501f48054d36f,7ec82b0fc3b4e215] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:51:20.996 INFO 4486 --- [prometheus-cache-monitor1,105501f48054d36f,7ec82b0fc3b4e215] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:51:21.311 INFO 4486 --- [ocp-async-6,61778cc47f73e8fc,bce4e009f5293bd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:21.564 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3a260f94576f03dd,5aa121c9a739018d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3a260f94576f03dd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:21.565 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3a260f94576f03dd,5aa121c9a739018d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3a260f94576f03dd, duration=1 ms] 2024-10-16 13:51:21.873 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b09abeb49d49cd3a,d699a6fe2d08f6db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b09abeb49d49cd3a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:21.873 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b09abeb49d49cd3a,d699a6fe2d08f6db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b09abeb49d49cd3a, duration=1 ms] 2024-10-16 13:51:22.291 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:22.994 INFO 4486 --- [prometheus-minute-schedule-1,76474ec017cf2a10,44af6ad9a3b1f298] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:23.000 INFO 4486 --- [prometheus-minute-schedule-1,76474ec017cf2a10,44af6ad9a3b1f298] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:23.012 INFO 4486 --- [ocp-async-5,24451c6f31892a29,81db0bedd910b184] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:51:23.312 INFO 4486 --- [ocp-async-1,553c6889118445df,e248081377d5b4ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:23.565 INFO 4486 --- [ocp-schedules-26,a231338ed273f338,4acb092a9ef49c97] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:51:23.930 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d2df0b039014a2ee,1ea341d0e9c8f1ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=d2df0b039014a2ee, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:51:23.930 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,d2df0b039014a2ee,1ea341d0e9c8f1ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=d2df0b039014a2ee, duration=5 ms] 2024-10-16 13:51:24.876 INFO 4486 --- [ocp-schedules-3,1eff3599999ebc26,042e8ed11a959574] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:51:25.287 INFO 4486 --- [ocp-schedules-16,e2c569d170d44379,9d9741e01e049547] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:51:25.287 INFO 4486 --- [ocp-schedules-16,390897308af7777d,4698821d6ccd3e13] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:51:25.289 INFO 4486 --- [ocp-schedules-16,390897308af7777d,4698821d6ccd3e13] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:25.292 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:25.312 INFO 4486 --- [ocp-async-9,9c90e375cd62faf2,93b82d54c1dfd6ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:25.317 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.318 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:25.327 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.329 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:25.352 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.353 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.354 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:51:25.355 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.356 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:25.384 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.386 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.386 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:25.389 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:51:25.389 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.392 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:51:25.393 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:25.401 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.401 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:25.412 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.421 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.422 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.423 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:25.423 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:51:25.428 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:51:25.433 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:25.435 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:25.462 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:51:25.468 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.470 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.471 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:51:25.480 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:51:25.480 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:51:25.522 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.524 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.524 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:25.528 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.531 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:51:25.534 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:25.536 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:51:25.545 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.546 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.547 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:51:25.556 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:51:25.556 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:51:25.562 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.562 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:25.591 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.591 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:25.622 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.622 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:25.650 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.650 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:25.679 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.680 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:25.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:25.741 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.743 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.744 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:25.748 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:51:25.753 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:51:25.758 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:51:25.759 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:51:25.783 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:51:25.929 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:51:25.960 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:25.962 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:25.963 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:51:25.964 INFO 4486 --- [ocp-schedules-4,2067b05415170a4f,b31d884dc3f2c9e9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:51:25.964 INFO 4486 --- [ocp-schedules-4,2067b05415170a4f,b31d884dc3f2c9e9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:51:25.994 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:51:25.994 INFO 4486 --- [prometheus-minute-schedule-1,3d8b31e2c6006a71,f28509a5e5ad7fd8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:25.994 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:51:26.000 INFO 4486 --- [prometheus-minute-schedule-1,3d8b31e2c6006a71,f28509a5e5ad7fd8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:26.002 INFO 4486 --- [Collect all tenants major compactions2,2067b05415170a4f,88c7dce92572d305] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:51:26.002 INFO 4486 --- [Collect all tenants major compactions1,2067b05415170a4f,e45285c99c0b0fd9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:51:26.002 INFO 4486 --- [Collect all tenants major compactions4,2067b05415170a4f,689a9e3a2cea4163] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:51:26.003 INFO 4486 --- [Collect all tenants major compactions5,2067b05415170a4f,55dd222c2752c0cb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:51:26.003 INFO 4486 --- [Collect all tenants major compactions3,2067b05415170a4f,df5326212f039da1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:51:26.060 INFO 4486 --- [Collect all tenants major compactions1,2067b05415170a4f,e45285c99c0b0fd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:26.060 INFO 4486 --- [Collect all tenants major compactions2,2067b05415170a4f,88c7dce92572d305] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:26.062 INFO 4486 --- [Collect all tenants major compactions1,2067b05415170a4f,e45285c99c0b0fd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:51:26.062 INFO 4486 --- [Collect all tenants major compactions2,2067b05415170a4f,88c7dce92572d305] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:51:26.065 INFO 4486 --- [Collect all tenants major compactions1,2067b05415170a4f,e45285c99c0b0fd9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:51:26.065 INFO 4486 --- [Collect all tenants major compactions2,2067b05415170a4f,88c7dce92572d305] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:51:26.065 INFO 4486 --- [Collect all tenants major compactions1,2067b05415170a4f,e45285c99c0b0fd9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:51:26.065 INFO 4486 --- [Collect all tenants major compactions2,2067b05415170a4f,88c7dce92572d305] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:51:26.069 INFO 4486 --- [Collect all tenants major compactions3,2067b05415170a4f,df5326212f039da1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:26.070 INFO 4486 --- [Collect all tenants major compactions3,2067b05415170a4f,df5326212f039da1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:51:26.074 INFO 4486 --- [Collect all tenants major compactions3,2067b05415170a4f,df5326212f039da1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:51:26.074 INFO 4486 --- [Collect all tenants major compactions3,2067b05415170a4f,df5326212f039da1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:51:26.079 INFO 4486 --- [Collect all tenants major compactions4,2067b05415170a4f,689a9e3a2cea4163] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:26.079 INFO 4486 --- [Collect all tenants major compactions5,2067b05415170a4f,55dd222c2752c0cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:26.080 INFO 4486 --- [Collect all tenants major compactions4,2067b05415170a4f,689a9e3a2cea4163] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:51:26.081 INFO 4486 --- [Collect all tenants major compactions5,2067b05415170a4f,55dd222c2752c0cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:51:26.084 INFO 4486 --- [Collect all tenants major compactions4,2067b05415170a4f,689a9e3a2cea4163] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:51:26.084 INFO 4486 --- [Collect all tenants major compactions4,2067b05415170a4f,689a9e3a2cea4163] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:51:26.086 INFO 4486 --- [Collect all tenants major compactions5,2067b05415170a4f,55dd222c2752c0cb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:51:26.086 INFO 4486 --- [Collect all tenants major compactions5,2067b05415170a4f,55dd222c2752c0cb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:51:26.087 INFO 4486 --- [ocp-schedules-4,2067b05415170a4f,b31d884dc3f2c9e9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:51:27.312 INFO 4486 --- [ocp-async-3,a641d793f2475713,a3110ef90c3e29eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:27.381 INFO 4486 --- [ocp-schedules-21,1f8dec7b2804b4c5,e6d21cc30e1a30d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:51:27.524 INFO 4486 --- [ocp-schedules-15,1c5e39da9b05ca39,722143ee77e8525a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:51:27.524+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:51:27.525 INFO 4486 --- [ocp-schedules-15,1c5e39da9b05ca39,722143ee77e8525a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:51:27.526 INFO 4486 --- [ocp-schedules-15,1c5e39da9b05ca39,722143ee77e8525a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:51:27.527 INFO 4486 --- [ocp-schedules-15,1c5e39da9b05ca39,722143ee77e8525a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:51:27.955 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9a5ced2f808f7455,0632d0adff57e25b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9a5ced2f808f7455, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:27.955 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9a5ced2f808f7455,0632d0adff57e25b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9a5ced2f808f7455, duration=1 ms] 2024-10-16 13:51:28.294 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:28.994 INFO 4486 --- [prometheus-minute-schedule-1,db4fc3cef71379b3,5e09ed2d49708db9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:29.004 INFO 4486 --- [prometheus-minute-schedule-1,db4fc3cef71379b3,5e09ed2d49708db9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:29.312 INFO 4486 --- [ocp-async-4,ea16a72642fd8f24,43e9a3b9ca3bbf35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:30.051 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2763c656c7a9191e,d06d8065aa2bf237] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2763c656c7a9191e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:30.051 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2763c656c7a9191e,d06d8065aa2bf237] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2763c656c7a9191e, duration=2 ms] 2024-10-16 13:51:30.103 INFO 4486 --- [prometheus-second-schedule-1,6c0a460a3bce7a7d,3637078a651a02a6] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:51:30.284 INFO 4486 --- [ocp-schedules-8,2b343264d99383d4,e991faf5244fbd94] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:51:30.299 WARN 4486 --- [scanner-13295204-1,2b343264d99383d4,3ac4e79a9c762975] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:51:30.403 INFO 4486 --- [alarm-detect-10,2b343264d99383d4,aaf68c2ce5fec12e] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057860 ~ 1729057880 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057875, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057875, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057875=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:51:30.934 INFO 4486 --- [ocp-schedules-11,7853040cd7fdaee6,515a0d6bfefa7532] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:51:31.295 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:31.312 INFO 4486 --- [ocp-async-2,5b44f86dd5dc82f7,f4ee6486b54bb68f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:31.568 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,48591de15267edf9,7466730d6d2264d5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=48591de15267edf9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:31.569 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,48591de15267edf9,7466730d6d2264d5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=48591de15267edf9, duration=1 ms] 2024-10-16 13:51:31.879 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,938230f12e7c3551,75a818f1949f8083] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=938230f12e7c3551, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:31.879 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,938230f12e7c3551,75a818f1949f8083] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=938230f12e7c3551, duration=1 ms] 2024-10-16 13:51:31.994 INFO 4486 --- [prometheus-minute-schedule-1,04cf2057efe657fe,70e4cc2a387852f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:32.000 INFO 4486 --- [prometheus-minute-schedule-1,04cf2057efe657fe,70e4cc2a387852f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:33.013 INFO 4486 --- [ocp-async-1,34250a8b2736eab2,8875988d130fa742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:51:33.313 INFO 4486 --- [ocp-async-5,1c815382325b85be,e79955264deb8ab8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:34.297 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:34.994 INFO 4486 --- [prometheus-minute-schedule-1,c60c9042e75022ed,61dd18a856da8921] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:35.000 INFO 4486 --- [prometheus-minute-schedule-1,c60c9042e75022ed,61dd18a856da8921] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:35.313 INFO 4486 --- [ocp-async-4,8da50557eebddfc4,2b834d16d31693c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:37.063 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:51:37.064 INFO 4486 --- [ocp-schedules-17,a05ecd871b8c05ba,95dcf45cba76b90f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:51:37.066 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:51:37.067 INFO 4486 --- [ocp-schedules-25,cbbd455bb461ad41,b803919ebd13ccb0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:51:37.068 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:51:37.068 INFO 4486 --- [ocp-schedules-25,cbbd455bb461ad41,b803919ebd13ccb0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:37.069 INFO 4486 --- [ocp-schedules-17,a05ecd871b8c05ba,95dcf45cba76b90f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:51:37.081 INFO 4486 --- [ocp-schedules-2,225cd6ad47b2e4e8,6e9bb07b543047ae] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:51:37.083 INFO 4486 --- [ocp-schedules-2,225cd6ad47b2e4e8,6e9bb07b543047ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:37.083 INFO 4486 --- [ocp-schedules-28,61310e9ce291e69d,7974d74b7225febb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 13:51:37.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:51:37.085 INFO 4486 --- [ocp-schedules-28,61310e9ce291e69d,7974d74b7225febb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:37.090 INFO 4486 --- [ocp-schedules-18,c584be03b07d9198,cf802c54241cbc4d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:51:37.090 INFO 4486 --- [ocp-schedules-12,0eeffcdd05e7c1de,488684984b9de01f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:51:37.091 INFO 4486 --- [ocp-schedules-30,5324e09d53e07cc7,c7f0b8b52790f5c3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:51:37.091 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:51:37.091 INFO 4486 --- [host-check-clockdiff1,c584be03b07d9198,ad436be084fb0d24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:51:37.092 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:51:37.092 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:51:37.092 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:51:37.094 INFO 4486 --- [host-check-clockdiff2,c584be03b07d9198,d7bb6ecd345139bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:51:37.094 INFO 4486 --- [host-check-clockdiff3,c584be03b07d9198,e7fb31cf77a05f2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:51:37.096 INFO 4486 --- [host-check-clockdiff4,c584be03b07d9198,5488305ae4f7954d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:51:37.097 INFO 4486 --- [host-check-agent-status1,5324e09d53e07cc7,5ff86f7ed7cbfff5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:51:37.098 INFO 4486 --- [host-check-agent-status2,5324e09d53e07cc7,af6ea97517aa784a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:51:37.099 INFO 4486 --- [host-check-agent-status4,5324e09d53e07cc7,ebe614867d3cd8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:51:37.098 INFO 4486 --- [host-check-agent-status3,5324e09d53e07cc7,95ad379c3008dac3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:51:37.103 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:37.105 INFO 4486 --- [host-check-agent-status2,5324e09d53e07cc7,af6ea97517aa784a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:51:37.103 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:51:37.103 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:51:37.103 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:51:37.119 INFO 4486 --- [host-check-agent-status1,5324e09d53e07cc7,5ff86f7ed7cbfff5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:37.119 INFO 4486 --- [host-check-agent-status4,5324e09d53e07cc7,ebe614867d3cd8b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:51:37.119 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:37.119 INFO 4486 --- [host-check-agent-status2,5324e09d53e07cc7,af6ea97517aa784a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:51:37.123 INFO 4486 --- [host-check-clockdiff1,c584be03b07d9198,ad436be084fb0d24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057897 0 0 , error= 2024-10-16 13:51:37.129 INFO 4486 --- [host-check-agent-status1,5324e09d53e07cc7,5ff86f7ed7cbfff5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:37.129 INFO 4486 --- [host-check-agent-status4,5324e09d53e07cc7,ebe614867d3cd8b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:51:37.129 INFO 4486 --- [host-check-agent-status3,5324e09d53e07cc7,95ad379c3008dac3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:51:37.141 INFO 4486 --- [host-check-clockdiff4,c584be03b07d9198,5488305ae4f7954d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057897 4 4 , error= 2024-10-16 13:51:37.144 INFO 4486 --- [host-check-clockdiff2,c584be03b07d9198,d7bb6ecd345139bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057897 4 4 , error= 2024-10-16 13:51:37.145 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.145 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:51:37.146 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:51:37.146 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:51:37.146 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:37.147 INFO 4486 --- [host-check-agent-status3,5324e09d53e07cc7,95ad379c3008dac3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:51:37.147 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:51:37.149 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.149 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:51:37.167 INFO 4486 --- [host-check-agent-status2,5324e09d53e07cc7,af6ea97517aa784a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:51:37.169 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.169 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:37.170 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,508e677ae8b32c1a,c69c8a28947bb80f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=508e677ae8b32c1a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 13:51:37.170 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:51:37.170 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,508e677ae8b32c1a,c69c8a28947bb80f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=508e677ae8b32c1a, duration=3 ms] 2024-10-16 13:51:37.170 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.171 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:51:37.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:37.172 INFO 4486 --- [host-check-agent-status4,5324e09d53e07cc7,ebe614867d3cd8b2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:51:37.173 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:51:37.173 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.174 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:51:37.173399987+08:00 2024-10-16 13:51:37.174 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:51:37.175 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.175 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:51:37.175 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:51:37.175 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:51:37.179522355+08:00 2024-10-16 13:51:37.175 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:51:37.174+08:00 2024-10-16 13:51:37.175 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.176 INFO 4486 --- [host-check-agent-status2,5324e09d53e07cc7,af6ea97517aa784a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:51:37.176 INFO 4486 --- [host-check-agent-status2,5324e09d53e07cc7,af6ea97517aa784a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:51:37.176 INFO 4486 --- [host-check-agent-status2,5324e09d53e07cc7,af6ea97517aa784a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:51:37.176 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:51:37.176 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.176 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:51:37.175+08:00 2024-10-16 13:51:37.176 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:51:37.177 INFO 4486 --- [host-check-agent-status1,5324e09d53e07cc7,5ff86f7ed7cbfff5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:51:37.177 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:51:37.177 INFO 4486 --- [host-check-agent-status4,5324e09d53e07cc7,ebe614867d3cd8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:51:37.177 INFO 4486 --- [host-check-agent-status4,5324e09d53e07cc7,ebe614867d3cd8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:51:37.178 INFO 4486 --- [host-check-agent-status4,5324e09d53e07cc7,ebe614867d3cd8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:51:37.179 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:51:37.182856692+08:00 2024-10-16 13:51:37.179 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:51:37.174+08:00 and status AVAILABLE 2024-10-16 13:51:37.179 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:51:37.175+08:00 and status AVAILABLE 2024-10-16 13:51:37.179 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:51:37.179+08:00 2024-10-16 13:51:37.180 INFO 4486 --- [host-check-agent-status3,5324e09d53e07cc7,95ad379c3008dac3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:51:37.182 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:51:37.179+08:00 and status AVAILABLE 2024-10-16 13:51:37.182 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:51:37.182 INFO 4486 --- [host-check-agent-status1,5324e09d53e07cc7,5ff86f7ed7cbfff5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:51:37.182 INFO 4486 --- [host-check-agent-status1,5324e09d53e07cc7,5ff86f7ed7cbfff5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:51:37.182 INFO 4486 --- [host-check-agent-status1,5324e09d53e07cc7,5ff86f7ed7cbfff5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:51:37.182 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:51:37.182 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:51:37.183 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.183 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:51:37.184 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:51:37.188050036+08:00 2024-10-16 13:51:37.184 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:51:37.184 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.185 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:51:37.184+08:00 2024-10-16 13:51:37.186 INFO 4486 --- [host-check-agent-status3,5324e09d53e07cc7,95ad379c3008dac3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:51:37.186 INFO 4486 --- [host-check-agent-status3,5324e09d53e07cc7,95ad379c3008dac3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:51:37.186 INFO 4486 --- [host-check-agent-status3,5324e09d53e07cc7,95ad379c3008dac3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:51:37.187 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.187 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:51:37.184+08:00 and status AVAILABLE 2024-10-16 13:51:37.187 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 13:51:37.187 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.187 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.187 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.187 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.187 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.187 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:51:37.188 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.189 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:51:37.191 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:37.192 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:51:37.192 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:51:37.192 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.193 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:51:37.193 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:37.193 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:51:37.193 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:51:37.198 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:51:37.199 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:51:37.199 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.203 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 13:51:37.203 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.204 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:51:37.205 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:51:37.205 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.206 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.207 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:51:37.209 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 13:51:37.212 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.215 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:37.216 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 13:51:37.216 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.217 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:37.218 INFO 4486 --- [ocp-schedules-3,00086828f313c2d2,b7aa774b92b94751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:51:37.219 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:37.219 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:51:37.247 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:51:37.248 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:51:37.255 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:51:37.259 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:51:37.259 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:51:37.265 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:51:37.270 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:51:37.277 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:37.279 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:37.283 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:37.284 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:51:37.297 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:37.303 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:51:37.307 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:51:37.312 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:51:37.313 INFO 4486 --- [ocp-async-10,9c97e3b1001f1ebb,215322c998a0b79a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:37.319 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:51:37.325 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:51:37.337 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:37.338 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:51:37.342 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:37.350 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:51:37.362 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:51:37.365 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:51:37.369 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:37.372 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:51:37.375 INFO 4486 --- [host-check-host-status3,0eeffcdd05e7c1de,66b2feca039f2950] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:51:37.375 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:51:37.386 INFO 4486 --- [host-check-host-status4,0eeffcdd05e7c1de,2e317bb2955a24d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:51:37.390 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:51:37.410 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:51:37.418 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:37.420 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.421 INFO 4486 --- [host-check-host-status2,0eeffcdd05e7c1de,f32739c35868e144] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:51:37.421 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:51:37.422 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.423 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.424 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:51:37.435 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.435 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.435 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.435 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.435 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.435 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.443 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:51:37.455 INFO 4486 --- [host-check-host-status1,0eeffcdd05e7c1de,a47fbd9b280ae127] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:51:37.463 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:51:37.494 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:51:37.539 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:37.556 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:37.558 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.558 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:51:37.559 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:51:37.560 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.561 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:51:37.563 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.563 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.568 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:51:37.577 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.577 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.577 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.577 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.577 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.577 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.584 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:51:37.770 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:51:37.814 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:37.817 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.818 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:51:37.819 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.820 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.820 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:51:37.830 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.830 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.830 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.830 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.830 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:37.831 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:37.852 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:51:37.960 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,12e99eb75647c282,3dc6d502f29020bb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=12e99eb75647c282, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:37.960 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,12e99eb75647c282,3dc6d502f29020bb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=12e99eb75647c282, duration=1 ms] 2024-10-16 13:51:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:51:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:51:37.994 INFO 4486 --- [prometheus-minute-schedule-1,e3bc71d2aa1569bc,82c33d05961700d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:37.997 INFO 4486 --- [ocp-schedules-6,0ee99615762fece0,9f04e379ddfe765b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 13:51:37.997 INFO 4486 --- [ocp-schedules-6,0ee99615762fece0,9f04e379ddfe765b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 13:51:38.001 INFO 4486 --- [prometheus-minute-schedule-1,e3bc71d2aa1569bc,82c33d05961700d4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:38.031 INFO 4486 --- [pool-10-thread-9,7d649f016c6b79e5,51f9c99989699f1f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 13:51:38.031 INFO 4486 --- [pool-10-thread-12,7d649f016c6b79e5,82c065a09056f14e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 13:51:38.032 INFO 4486 --- [pool-10-thread-11,7d649f016c6b79e5,f710c6c73d6ebce9] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 13:51:38.036 INFO 4486 --- [pool-10-thread-10,7d649f016c6b79e5,e7a9d05070bd8b22] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 13:51:38.037 INFO 4486 --- [ForkJoinPool-6-worker-15,7d649f016c6b79e5,c8c0dc1dfd29d8bd] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 13:51:38.037 INFO 4486 --- [ForkJoinPool-6-worker-15,7d649f016c6b79e5,c8c0dc1dfd29d8bd] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 13:51:38.037 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 13:51:38.037 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 13:51:38.037 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 13:51:38.037 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 13:51:38.041 INFO 4486 --- [pool-10-thread-10,7d649f016c6b79e5,e7a9d05070bd8b22] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 13:51:38.041 INFO 4486 --- [pool-10-thread-10,7d649f016c6b79e5,e7a9d05070bd8b22] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 13:51:38.044 INFO 4486 --- [pool-overview-container-5,63f2824e788631d1,d5ae66549cf5f7a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.047 INFO 4486 --- [pool-overview-container-5,63f2824e788631d1,d5ae66549cf5f7a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:51:38.070 INFO 4486 --- [host-check-clockdiff3,c584be03b07d9198,e7fb31cf77a05f2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057898 4 4 , error= 2024-10-16 13:51:38.075 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:51:38.075 INFO 4486 --- [ForkJoinPool-6-worker-15,7d649f016c6b79e5,c8c0dc1dfd29d8bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.076 INFO 4486 --- [ForkJoinPool-6-worker-15,7d649f016c6b79e5,c8c0dc1dfd29d8bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:51:38.077 INFO 4486 --- [ForkJoinPool-6-worker-15,7d649f016c6b79e5,c8c0dc1dfd29d8bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.078 INFO 4486 --- [pool-10-thread-9,7d649f016c6b79e5,51f9c99989699f1f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 13:51:38.078 INFO 4486 --- [ForkJoinPool-6-worker-15,7d649f016c6b79e5,c8c0dc1dfd29d8bd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:51:38.090 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.091 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:51:38.097 WARN 4486 --- [ForkJoinPool-6-worker-15,7d649f016c6b79e5,c8c0dc1dfd29d8bd] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:51:38.099 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.099 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.100 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:51:38.101 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:51:38.102 INFO 4486 --- [pool-overview-container-5,63f2824e788631d1,d5ae66549cf5f7a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.103 INFO 4486 --- [pool-10-thread-9,7d649f016c6b79e5,51f9c99989699f1f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 13:51:38.104 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.105 INFO 4486 --- [pool-overview-container-5,63f2824e788631d1,d5ae66549cf5f7a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:51:38.105 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:51:38.106 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.108 INFO 4486 --- [ForkJoinPool-6-worker-10,7d649f016c6b79e5,31e7328eb496123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.110 INFO 4486 --- [pool-10-thread-9,7d649f016c6b79e5,51f9c99989699f1f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 13:51:38.110 INFO 4486 --- [ForkJoinPool-6-worker-10,7d649f016c6b79e5,31e7328eb496123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:51:38.113 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:51:38.114 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.116 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:51:38.122 INFO 4486 --- [ForkJoinPool-6-worker-10,7d649f016c6b79e5,31e7328eb496123d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 13:51:38.123 INFO 4486 --- [ForkJoinPool-6-worker-10,7d649f016c6b79e5,31e7328eb496123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.124 INFO 4486 --- [ForkJoinPool-6-worker-10,7d649f016c6b79e5,31e7328eb496123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:51:38.125 INFO 4486 --- [ForkJoinPool-6-worker-10,7d649f016c6b79e5,31e7328eb496123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.126 INFO 4486 --- [ForkJoinPool-6-worker-10,7d649f016c6b79e5,31e7328eb496123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:51:38.127 INFO 4486 --- [ForkJoinPool-6-worker-3,7d649f016c6b79e5,63159f50554f5b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.130 INFO 4486 --- [ForkJoinPool-6-worker-3,7d649f016c6b79e5,63159f50554f5b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:51:38.136 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:38.138 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.139 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:51:38.139 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:51:38.140 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.140 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:38.141 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.141 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:38.141 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.142 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:51:38.143 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:51:38.147 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:51:38.147 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.147 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.149 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:51:38.153 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:51:38.154 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:38.154 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:38.154 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:38.154 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:38.154 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:51:38.155 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.155 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.156 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:51:38.157 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.158 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:51:38.160 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.161 INFO 4486 --- [ForkJoinPool-6-worker-3,7d649f016c6b79e5,63159f50554f5b4f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:51:38.161 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:51:38.162 INFO 4486 --- [ForkJoinPool-6-worker-3,7d649f016c6b79e5,63159f50554f5b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.164 INFO 4486 --- [ForkJoinPool-6-worker-3,7d649f016c6b79e5,63159f50554f5b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:51:38.168 INFO 4486 --- [ForkJoinPool-6-worker-3,7d649f016c6b79e5,63159f50554f5b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.170 INFO 4486 --- [ForkJoinPool-6-worker-3,7d649f016c6b79e5,63159f50554f5b4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:51:38.172 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.178 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:38.180 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.180 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:51:38.184 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.186 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:51:38.186 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:51:38.196 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.197 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:51:38.201 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:51:38.203 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.203 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.204 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.204 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:51:38.205 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:51:38.205 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:51:38.207 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.208 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:51:38.208 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 13:51:38.210 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.210 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:38.211 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:51:38.212 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.216 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 13:51:38.217 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:51:38.224 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:51:38.227 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:51:38.229 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.232 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.235 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.235 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:51:38, 2024-10-16 13:51:38] 2024-10-16 13:51:38.237 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 13:51:38.237 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:51:38.245 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 13:51:38.246 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.247 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.248 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:51:38, 2024-10-16 13:51:38] 2024-10-16 13:51:38.252 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:51:38.256 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:51:38.256 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:51:38.257 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.257 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.259 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:51:38, 2024-10-16 13:51:38] 2024-10-16 13:51:38.259 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:51:38.262 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.264 INFO 4486 --- [ForkJoinPool-6-worker-1,7d649f016c6b79e5,9456aaff24565d9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:51:38.268 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:38.269 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.270 INFO 4486 --- [pool-10-thread-11,7d649f016c6b79e5,f710c6c73d6ebce9] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 13:51:38.275 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:51:38.276 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 13:51:38.276 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:51:38.278 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.278 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.279 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:51:38, 2024-10-16 13:51:38] 2024-10-16 13:51:38.279 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:51:38, 2024-10-16 13:51:38] 2024-10-16 13:51:38.282 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 13:51:38.282 INFO 4486 --- [ForkJoinPool-6-worker-6,7d649f016c6b79e5,f6043fa9670a474f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 13:51:38.287 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:38.288 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.295 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 13:51:38.310 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:51:38.310 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.311 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:38.313 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:51:38, 2024-10-16 13:51:38] 2024-10-16 13:51:38.316 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:51:38.316 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 13:51:38.316 INFO 4486 --- [ForkJoinPool-6-worker-8,7d649f016c6b79e5,f2b30e1965832eed] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 13:51:38.319 INFO 4486 --- [pool-10-thread-12,7d649f016c6b79e5,82c065a09056f14e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 13:51:38.466 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:51:38.232052, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:51:38.466 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:51:38.232052, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:51:38.475 INFO 4486 --- [ocp-schedules-30,c744c01f7149d422,066468505e9660e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:51:38.475 INFO 4486 --- [ocp-schedules-30,c744c01f7149d422,066468505e9660e9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:51:38.514 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:51:38.232052} } 2024-10-16 13:51:38.577 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:51:38.584 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:51:38.587 INFO 4486 --- [ocp-schedules-27,33188c150b9d1f57,f1f06aabd3460c60] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:51:39.123 INFO 4486 --- [ocp-schedules-22,2b74caa1749dd036,cc84bb5c2ca00c82] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:51:39.313 INFO 4486 --- [ocp-async-6,07e36da57b05fc14,60bf0a66faeb27f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:40.006 INFO 4486 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:51:40.007 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:51:40.024 INFO 4486 --- [Check obproxy1,cf930ceb7468dbb0,9b6a2053ec4b8be7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:51:40.025 INFO 4486 --- [Check obproxy1,cf930ceb7468dbb0,9b6a2053ec4b8be7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:40.025 INFO 4486 --- [Check obproxy2,abc01506ac132fbe,e387bab92141fbaa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:51:40.025 INFO 4486 --- [Check obproxy2,abc01506ac132fbe,e387bab92141fbaa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:40.028 INFO 4486 --- [Collect obproxy2,bfa6e08d4793113e,d041754cd4d3f9ca] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:51:40.028 INFO 4486 --- [Collect obproxy2,bfa6e08d4793113e,d041754cd4d3f9ca] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:40.028 INFO 4486 --- [Collect obproxy1,a19740fcfd7f332c,2029f2cb008e9687] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:51:40.028 INFO 4486 --- [Collect obproxy1,a19740fcfd7f332c,2029f2cb008e9687] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:51:40.053 INFO 4486 --- [Check obproxy1,cf930ceb7468dbb0,9b6a2053ec4b8be7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:51:40.053 INFO 4486 --- [Check obproxy1,cf930ceb7468dbb0,9b6a2053ec4b8be7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:51:40.053 INFO 4486 --- [Check obproxy2,abc01506ac132fbe,e387bab92141fbaa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:51:40.053 INFO 4486 --- [Check obproxy2,abc01506ac132fbe,e387bab92141fbaa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:51:40.053 INFO 4486 --- [Collect obproxy2,bfa6e08d4793113e,d041754cd4d3f9ca] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:51:40.056 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,bb61420467ea560b,26a6b17688ca683e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bb61420467ea560b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:40.056 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,bb61420467ea560b,26a6b17688ca683e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bb61420467ea560b, duration=1 ms] 2024-10-16 13:51:40.059 INFO 4486 --- [Collect obproxy2,bfa6e08d4793113e,d041754cd4d3f9ca] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:51:40.059 INFO 4486 --- [Collect obproxy1,a19740fcfd7f332c,2029f2cb008e9687] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:51:40.064 INFO 4486 --- [Collect obproxy1,a19740fcfd7f332c,2029f2cb008e9687] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:51:40.065 INFO 4486 --- [Collect obproxy2,bfa6e08d4793113e,d041754cd4d3f9ca] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:51:40.069 INFO 4486 --- [Collect obproxy1,a19740fcfd7f332c,2029f2cb008e9687] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:51:40.298 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:40.464 INFO 4486 --- [ocp-schedules-23,ca97a5b387372785,b8454a016775248b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:51:40.498 WARN 4486 --- [scanner-13295557-1,ca97a5b387372785,46102ef4acc9ec1a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:51:40.552 INFO 4486 --- [alarm-detect-7,ca97a5b387372785,a47bd904dea95fef] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057875 ~ 1729057890 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057890, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057890, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057890=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:51:40.943 INFO 4486 --- [ocp-schedules-6,5d452ed5e519332a,73cd8392555ed389] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:51:40.994 INFO 4486 --- [prometheus-minute-schedule-1,e8f1ac4338e5654b,578348645074b4e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:40.995 INFO 4486 --- [prometheus-hour-schedule-1,76e02875b667f098,c3e7b2dda8bd68d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:51:41.000 INFO 4486 --- [prometheus-minute-schedule-1,e8f1ac4338e5654b,578348645074b4e7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:41.313 INFO 4486 --- [ocp-async-4,09ccb40b9e015c77,8b4b0f5631c613d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:41.573 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,4feed676be10f778,3ff96cae08082d64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4feed676be10f778, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:41.574 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,4feed676be10f778,3ff96cae08082d64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4feed676be10f778, duration=2 ms] 2024-10-16 13:51:41.883 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a330a45dd808736a,e6595d3f05d35c0c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a330a45dd808736a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:41.883 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a330a45dd808736a,e6595d3f05d35c0c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a330a45dd808736a, duration=1 ms] 2024-10-16 13:51:42.144 INFO 4486 --- [ocp-schedules-14,f2271c3950d6f0dc,c28419d3585bf713] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:51:43.013 INFO 4486 --- [ocp-async-8,de8a4c6f268889cd,930aa15270f31f1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:51:43.300 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:43.314 INFO 4486 --- [ocp-async-9,ec85ad88b92c974a,587af594ba32817f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:43.994 INFO 4486 --- [prometheus-minute-schedule-1,b56ca4e8889e6035,0035c0dd93370df6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:44.001 INFO 4486 --- [prometheus-minute-schedule-1,b56ca4e8889e6035,0035c0dd93370df6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:45.314 INFO 4486 --- [ocp-async-5,f553c72f2e4f975d,fcd65d7aeae4da9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:45.540 INFO 4486 --- [ocp-schedules-27,33fe13e21810338a,62f8edeab38f0bd1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:51:45.546 INFO 4486 --- [ocp-schedules-27,33fe13e21810338a,62f8edeab38f0bd1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:51:45.546 INFO 4486 --- [ocp-schedules-27,33fe13e21810338a,62f8edeab38f0bd1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:51:45.571 INFO 4486 --- [ocp-schedules-24,e6a582c8d6ce5d6b,668f8ffb359a3f7e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:51:45.571 INFO 4486 --- [ocp-schedules-24,e6a582c8d6ce5d6b,668f8ffb359a3f7e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:51:46.301 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:46.994 INFO 4486 --- [prometheus-minute-schedule-1,4932db3e15c02fd9,ffead0001cd92961] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:46.999 INFO 4486 --- [prometheus-minute-schedule-1,4932db3e15c02fd9,ffead0001cd92961] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:47.070 INFO 4486 --- [ocp-schedules-25,cbbd455bb461ad41,b803919ebd13ccb0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:51:47.177 INFO 4486 --- [ocp-schedules-2,225cd6ad47b2e4e8,6e9bb07b543047ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:51:47.207 INFO 4486 --- [ocp-schedules-9,8d1bf5e4364b891d,5000a4fd0a000b34] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:51:47.219 INFO 4486 --- [ocp-schedules-28,61310e9ce291e69d,7974d74b7225febb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 13:51:47.314 INFO 4486 --- [ocp-async-4,53cc3e9567832996,c3b6eec8404fa930] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:47.964 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ab77a310a57c534c,b4f3e6fce2884563] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ab77a310a57c534c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:47.965 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ab77a310a57c534c,b4f3e6fce2884563] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ab77a310a57c534c, duration=2 ms] 2024-10-16 13:51:48.759 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:48.760 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:51:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:51:48.763 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:51:49.302 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:49.314 INFO 4486 --- [ocp-async-9,b540d4d8bf5325bc,c2cf92229cc0c0ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:49.994 INFO 4486 --- [prometheus-minute-schedule-1,0a501f50c594d67e,94c3cf52f88dbdeb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:50.003 INFO 4486 --- [prometheus-minute-schedule-1,0a501f50c594d67e,94c3cf52f88dbdeb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:50.062 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0ec934c08870bf2f,0d2a55ab52c5c23f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0ec934c08870bf2f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:50.062 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0ec934c08870bf2f,0d2a55ab52c5c23f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0ec934c08870bf2f, duration=2 ms] 2024-10-16 13:51:50.501 INFO 4486 --- [ocp-schedules-24,cee918965cd27ddc,88dafe0011ae2f75] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:51:50.501 INFO 4486 --- [ocp-schedules-24,cee918965cd27ddc,88dafe0011ae2f75] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:51:50.502 INFO 4486 --- [ocp-schedules-24,cee918965cd27ddc,88dafe0011ae2f75] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:51:50.502 INFO 4486 --- [ocp-schedules-24,cee918965cd27ddc,88dafe0011ae2f75] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:51:50.502 INFO 4486 --- [ocp-schedules-24,cee918965cd27ddc,88dafe0011ae2f75] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:51:50.502 INFO 4486 --- [ocp-schedules-24,cee918965cd27ddc,88dafe0011ae2f75] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:51:50.617 INFO 4486 --- [ocp-schedules-23,16893d1c06746983,bcaa77d90f385c09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:51:50.722 WARN 4486 --- [scanner-13295843-1,16893d1c06746983,0053c1f131588ee4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:51:50.748 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:51:50.749 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:51:50.767 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:51:50.767 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:51:50.767 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:51:50.782 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:51:50.782 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:51:50.782 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:51:50.791 INFO 4486 --- [ocp-schedules-1,d0e196bd6efab179,c673db526ca95cfb] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:51:50.949 INFO 4486 --- [ocp-schedules-13,ff09a2670df06041,372c400f65459901] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:51:50.970 INFO 4486 --- [ocp-schedules-10,e6ccfda47bea1aae,8eeb9d0172a9a060] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:51:50.970 INFO 4486 --- [ocp-schedules-10,e6ccfda47bea1aae,8eeb9d0172a9a060] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:51:50.970 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:51:50.970 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:51:50.970 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:51:50.970 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:51:50.971 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:51:50.976 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:51:50.976 INFO 4486 --- [ocp-schedules-10,e6ccfda47bea1aae,8eeb9d0172a9a060] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:51:50.976 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:51:50.976 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:51:50.977 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:50.977 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:51:50.977 INFO 4486 --- [ocp-schedules-10,e6ccfda47bea1aae,8eeb9d0172a9a060] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:51:50.977 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:51:50.977 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:51:50.978 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:51:50.978 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:50.979 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:51:51.001 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:51:51.003 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:51:51.003 INFO 4486 --- [ocp-schedules-10,e6ccfda47bea1aae,8eeb9d0172a9a060] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:51:51.003 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:51:51.006 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:51:51.008 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:51:51.011 INFO 4486 --- [ocp-schedules-10,e6ccfda47bea1aae,8eeb9d0172a9a060] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:51:51.315 INFO 4486 --- [ocp-async-5,5ce199f30b812061,95d5f5b49265c69c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:51.579 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,f0df948cbdc7b94f,b6bb46085e0835ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f0df948cbdc7b94f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:51.579 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,f0df948cbdc7b94f,b6bb46085e0835ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f0df948cbdc7b94f, duration=1 ms] 2024-10-16 13:51:51.887 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,d171c45169c680bd,830d6bd9698fdbd3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d171c45169c680bd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:51.887 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,d171c45169c680bd,830d6bd9698fdbd3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d171c45169c680bd, duration=1 ms] 2024-10-16 13:51:52.303 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:52.377 INFO 4486 --- [ocp-schedules-24,1cca1d4e76141c96,f0c8562080ea4d0c] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:51:52.994 INFO 4486 --- [prometheus-minute-schedule-1,94dcecaa3daf1ba1,84e9461f618128ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:53.000 INFO 4486 --- [prometheus-minute-schedule-1,94dcecaa3daf1ba1,84e9461f618128ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:53.013 INFO 4486 --- [ocp-async-3,bc5fbba16038b6db,c4f17ed7c3378c92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:51:53.315 INFO 4486 --- [ocp-async-4,3648fc8285a58282,0aa5f4b62cd1427a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:53.543 INFO 4486 --- [ocp-schedules-24,a8efb5dcc07081d8,434c46c2ef5e7786] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:51:53.545 INFO 4486 --- [ocp-schedules-24,a8efb5dcc07081d8,434c46c2ef5e7786] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:53.545 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:51:53.545 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:51:53.545 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:51:55.304 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:55.315 INFO 4486 --- [ocp-async-9,6da279a30a07c608,f61cda548912c721] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:55.994 INFO 4486 --- [prometheus-minute-schedule-1,c42c0e7f7863c930,99847594489aab27] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:55.994 INFO 4486 --- [ocp-schedules-16,390897308af7777d,4698821d6ccd3e13] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:51:55.995 INFO 4486 --- [ocp-schedules-16,b3685c4fe3a1d1d9,0ee88496a621a4d2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:51:55.996 INFO 4486 --- [ocp-schedules-16,b3685c4fe3a1d1d9,0ee88496a621a4d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:51:56.001 INFO 4486 --- [prometheus-minute-schedule-1,c42c0e7f7863c930,99847594489aab27] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:56.026 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.027 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:56.036 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.037 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:56.061 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.062 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.062 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:51:56.062 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.063 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:56.091 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.091 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.092 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:56.093 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:51:56.093 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.096 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:51:56.098 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:56.107 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.107 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:56.119 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.129 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.129 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.130 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:56.131 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:51:56.136 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:51:56.140 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:51:56.142 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:51:56.167 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.168 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:51:56.168 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.169 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:51:56.202 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:51:56.202 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:51:56.221 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.222 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.223 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:51:56.227 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.230 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:51:56.233 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:56.255 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:51:56.260 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.260 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:56.264 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.266 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.266 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:51:56.276 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:51:56.276 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:51:56.292 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.292 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:56.317 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.317 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:56.342 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.343 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:51:56.368 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.368 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:51:56.397 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:51:56.426 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.428 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.429 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:51:56.434 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:51:56.439 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:51:56.443 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:51:56.445 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:51:56.467 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:51:56.602 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:51:56.630 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:51:56.631 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:51:56.632 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:51:56.661 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:51:56.661 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:51:57.315 INFO 4486 --- [ocp-async-5,92a1851cfa6be397,12f1aa1492f8655c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:57.973 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,fdc13aff15a3ae61,f23785384476cd9f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fdc13aff15a3ae61, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:51:57.974 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,fdc13aff15a3ae61,f23785384476cd9f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fdc13aff15a3ae61, duration=2 ms] 2024-10-16 13:51:58.306 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:51:58.994 INFO 4486 --- [prometheus-minute-schedule-1,3208514e64fa353e,106e5db2ae581acb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057920, interval=60 2024-10-16 13:51:59.000 INFO 4486 --- [prometheus-minute-schedule-1,3208514e64fa353e,106e5db2ae581acb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:51:59.127 INFO 4486 --- [ocp-schedules-15,2ba290cb70cf004b,78159e72e1f9caf9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:51:59.315 INFO 4486 --- [ocp-async-8,ec4a2bb6b68a0e84,1cdf1581e703b1be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:51:59.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,e1cafaec85b443eb,097ddf08ad0c7e38] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=e1cafaec85b443eb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:51:59.836 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,e1cafaec85b443eb,097ddf08ad0c7e38] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=e1cafaec85b443eb, duration=1 ms] 2024-10-16 13:52:00.068 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,861a08118c7bd4e5,3780af47571e1ffe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=861a08118c7bd4e5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:00.068 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,861a08118c7bd4e5,3780af47571e1ffe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=861a08118c7bd4e5, duration=1 ms] 2024-10-16 13:52:00.856 INFO 4486 --- [ocp-schedules-20,99ceba2475afc326,2bf3cbef59e26033] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:52:00.886 WARN 4486 --- [scanner-13296135-1,99ceba2475afc326,15026794df5ce206] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:52:00.948 INFO 4486 --- [alarm-detect-9,99ceba2475afc326,396897bdfd26fe18] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057890 ~ 1729057910 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057905, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057905, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057905=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:52:00.954 INFO 4486 --- [ocp-schedules-7,8d4c38dbc8beed43,85b865d650ebb037] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:52:01.307 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:01.315 INFO 4486 --- [ocp-async-4,fd024f67b7eb0a7e,317e0add380f726b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:01.584 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,808eaf2e336c6e26,9c52cab6108b0fe5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=808eaf2e336c6e26, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:01.584 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,808eaf2e336c6e26,9c52cab6108b0fe5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=808eaf2e336c6e26, duration=1 ms] 2024-10-16 13:52:01.893 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,55132c71fcc35fb9,c722fcb2c8dab2bb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=55132c71fcc35fb9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:01.893 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,55132c71fcc35fb9,c722fcb2c8dab2bb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=55132c71fcc35fb9, duration=1 ms] 2024-10-16 13:52:01.994 INFO 4486 --- [prometheus-minute-schedule-1,22cf7aaae522ba68,bfadc69f33899374] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:02.001 INFO 4486 --- [prometheus-minute-schedule-1,22cf7aaae522ba68,bfadc69f33899374] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:02.154 INFO 4486 --- [ocp-schedules-28,12508b28c13fa4f2,0b311930c61be72e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:52:02.154+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:52:02.158 INFO 4486 --- [ocp-schedules-28,12508b28c13fa4f2,0b311930c61be72e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:52:02.164 INFO 4486 --- [ocp-schedules-28,12508b28c13fa4f2,0b311930c61be72e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:52:02.181 INFO 4486 --- [ocp-schedules-28,12508b28c13fa4f2,0b311930c61be72e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:52:03.013 INFO 4486 --- [ocp-async-1,8b215fcfd3b820f8,54b593844f15180d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:52:03.315 INFO 4486 --- [ocp-async-10,73fb13e8faa77cc5,99ef61b385a4dabd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:03.546 INFO 4486 --- [ocp-schedules-24,a8efb5dcc07081d8,434c46c2ef5e7786] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:52:04.308 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:04.994 INFO 4486 --- [prometheus-minute-schedule-1,2d4f975a5e20ac13,a528dac0f0954502] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:05.001 INFO 4486 --- [prometheus-minute-schedule-1,2d4f975a5e20ac13,a528dac0f0954502] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:05.316 INFO 4486 --- [ocp-async-6,a0950d56e74d8a76,0ba3ccd0654f356d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:05.318 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,dda1e76eae6ef6be,3a06c19b1e939cb1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=dda1e76eae6ef6be, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:52:05.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,dda1e76eae6ef6be,3a06c19b1e939cb1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=dda1e76eae6ef6be, duration=3 ms] 2024-10-16 13:52:07.066 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:52:07.066 INFO 4486 --- [ocp-schedules-15,855a26e22da8b377,752a2e900f1c73fd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:52:07.068 INFO 4486 --- [ocp-schedules-15,855a26e22da8b377,752a2e900f1c73fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:07.081 INFO 4486 --- [ocp-schedules-20,42a0b83068585e77,c4caf6fb1e4825fe] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:52:07.083 INFO 4486 --- [ocp-schedules-20,42a0b83068585e77,c4caf6fb1e4825fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:52:07.089 INFO 4486 --- [ocp-schedules-13,98d69fbee496b4d5,faa830bcda5c07a6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:52:07.089 INFO 4486 --- [ocp-schedules-24,f43f21dc51177c5a,97b7257c12f24770] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:52:07.090 INFO 4486 --- [host-check-agent-status1,98d69fbee496b4d5,167ce585922ba03a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:52:07.090 INFO 4486 --- [host-check-agent-status2,98d69fbee496b4d5,29a0630db7e81b84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:52:07.090 INFO 4486 --- [ocp-schedules-28,651b83598fb1f360,ffefca154fc19568] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:52:07.090 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:52:07.091 INFO 4486 --- [host-check-agent-status3,98d69fbee496b4d5,ae588278a7a98b60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:52:07.091 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:52:07.091 INFO 4486 --- [host-check-agent-status4,98d69fbee496b4d5,617618fa53a56789] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:52:07.091 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:52:07.092 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:52:07.093 INFO 4486 --- [host-check-clockdiff2,651b83598fb1f360,3d55a0aa0c2b1129] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:52:07.093 INFO 4486 --- [host-check-clockdiff4,651b83598fb1f360,3c85488ec5fda25f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:52:07.094 INFO 4486 --- [host-check-clockdiff3,651b83598fb1f360,1b10e5b91535ec35] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:52:07.094 INFO 4486 --- [host-check-clockdiff1,651b83598fb1f360,6858f221eafe2725] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:52:07.098 INFO 4486 --- [host-check-agent-status1,98d69fbee496b4d5,167ce585922ba03a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:07.099 INFO 4486 --- [host-check-agent-status2,98d69fbee496b4d5,29a0630db7e81b84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:52:07.099 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:52:07.100 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:07.100 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:52:07.100 INFO 4486 --- [host-check-agent-status3,98d69fbee496b4d5,ae588278a7a98b60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:52:07.100 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:52:07.100 INFO 4486 --- [host-check-agent-status2,98d69fbee496b4d5,29a0630db7e81b84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:52:07.101 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:07.100 INFO 4486 --- [host-check-agent-status1,98d69fbee496b4d5,167ce585922ba03a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:07.118 INFO 4486 --- [host-check-agent-status3,98d69fbee496b4d5,ae588278a7a98b60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:52:07.118 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:07.125 INFO 4486 --- [host-check-clockdiff3,651b83598fb1f360,1b10e5b91535ec35] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057927 5 5 , error= 2024-10-16 13:52:07.125 INFO 4486 --- [host-check-clockdiff2,651b83598fb1f360,3d55a0aa0c2b1129] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057927 5 5 , error= 2024-10-16 13:52:07.125 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:07.126 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:52:07.130 INFO 4486 --- [host-check-agent-status4,98d69fbee496b4d5,617618fa53a56789] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:07.133 INFO 4486 --- [host-check-clockdiff1,651b83598fb1f360,6858f221eafe2725] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057927 0 0 , error= 2024-10-16 13:52:07.136 INFO 4486 --- [host-check-agent-status4,98d69fbee496b4d5,617618fa53a56789] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:07.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:52:07.160 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:52:07.160 INFO 4486 --- [host-check-agent-status3,98d69fbee496b4d5,ae588278a7a98b60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:52:07.160 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:52:07.164 INFO 4486 --- [host-check-agent-status1,98d69fbee496b4d5,167ce585922ba03a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:52:07.165 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:52:07.165 INFO 4486 --- [host-check-agent-status2,98d69fbee496b4d5,29a0630db7e81b84] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:52:07.168 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:52:07.169206046+08:00 2024-10-16 13:52:07.168 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:52:07.169403552+08:00 2024-10-16 13:52:07.168 INFO 4486 --- [host-check-agent-status3,98d69fbee496b4d5,ae588278a7a98b60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:52:07.168 INFO 4486 --- [host-check-agent-status3,98d69fbee496b4d5,ae588278a7a98b60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:52:07.168 INFO 4486 --- [host-check-agent-status3,98d69fbee496b4d5,ae588278a7a98b60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:52:07.169 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:52:07.168+08:00 2024-10-16 13:52:07.169 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:52:07.168+08:00 2024-10-16 13:52:07.169 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:52:07.170 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:52:07.174213527+08:00 2024-10-16 13:52:07.170 INFO 4486 --- [host-check-agent-status4,98d69fbee496b4d5,617618fa53a56789] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:52:07.171 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:52:07.170+08:00 2024-10-16 13:52:07.171 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:52:07.171250733+08:00 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:52:07.171+08:00 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-agent-status1,98d69fbee496b4d5,167ce585922ba03a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-agent-status1,98d69fbee496b4d5,167ce585922ba03a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-agent-status1,98d69fbee496b4d5,167ce585922ba03a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:52:07.168+08:00 and status AVAILABLE 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-agent-status2,98d69fbee496b4d5,29a0630db7e81b84] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-agent-status2,98d69fbee496b4d5,29a0630db7e81b84] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:52:07.172 INFO 4486 --- [host-check-agent-status2,98d69fbee496b4d5,29a0630db7e81b84] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:52:07.172 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.173 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:52:07.168+08:00 and status AVAILABLE 2024-10-16 13:52:07.173 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:07.174 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:52:07.170+08:00 and status AVAILABLE 2024-10-16 13:52:07.175 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:52:07.171+08:00 and status AVAILABLE 2024-10-16 13:52:07.175 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:52:07.175 INFO 4486 --- [host-check-agent-status4,98d69fbee496b4d5,617618fa53a56789] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:52:07.175 INFO 4486 --- [host-check-agent-status4,98d69fbee496b4d5,617618fa53a56789] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:52:07.175 INFO 4486 --- [host-check-agent-status4,98d69fbee496b4d5,617618fa53a56789] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:52:07.175 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:52:07.176 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:52:07.176 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:52:07.214 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:52:07.215 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.220 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:52:07.221 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:07.222 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:52:07.222 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:07.222 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:52:07.222 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:07.223 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:52:07.223 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:07.224 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:52:07.229 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.230 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:07.241 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.242 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:07.247 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:07.247 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:07.247 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:07.253 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:07.253 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.254 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:07.257 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.258 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:52:07.268 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:07.269 INFO 4486 --- [ocp-schedules-11,84965ed9d0f0bfa4,2c7ea88ef53c053c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:52:07.284 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:52:07.285 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:52:07.287 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:52:07.295 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:52:07.295 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:52:07.298 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:52:07.308 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:52:07.309 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:07.313 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:07.314 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:07.316 INFO 4486 --- [ocp-async-1,59b5287e3813b187,36d3ebd08348d8af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:07.317 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:07.319 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:52:07.336 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:07.345 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:52:07.351 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:52:07.354 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:52:07.357 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:52:07.360 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:52:07.368 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:52:07.373 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:07.376 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:07.385 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:07.392 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:52:07.401 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:52:07.403 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:52:07.412 INFO 4486 --- [host-check-host-status3,f43f21dc51177c5a,0d5fe6fbeceeac60] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:52:07.413 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:52:07.421 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:07.424 INFO 4486 --- [host-check-host-status4,f43f21dc51177c5a,1ba11c2d7824c66d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:52:07.426 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:52:07.437 INFO 4486 --- [host-check-host-status2,f43f21dc51177c5a,7985744656d523df] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:52:07.479 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:52:07.494 INFO 4486 --- [host-check-host-status1,f43f21dc51177c5a,f1bcfd3554bbd65a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:52:07.978 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,043906a3633989ad,d7ba0c3ddf5e7b1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=043906a3633989ad, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:07.978 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,043906a3633989ad,d7ba0c3ddf5e7b1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=043906a3633989ad, duration=1 ms] 2024-10-16 13:52:07.994 INFO 4486 --- [prometheus-minute-schedule-1,00e92b91d00f3688,4e302d011d7adf2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:08.000 INFO 4486 --- [prometheus-minute-schedule-1,00e92b91d00f3688,4e302d011d7adf2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:08.058 INFO 4486 --- [host-check-clockdiff4,651b83598fb1f360,3c85488ec5fda25f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057928 4 4 , error= 2024-10-16 13:52:09.316 INFO 4486 --- [ocp-async-9,561977c4326dcea2,a93d6cea634bf849] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:10.006 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:52:10.006 INFO 4486 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:52:10.023 INFO 4486 --- [Check obproxy2,960784f256016973,c6f9ab19431d7aac] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:52:10.023 INFO 4486 --- [Check obproxy2,960784f256016973,c6f9ab19431d7aac] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:10.024 INFO 4486 --- [Check obproxy1,328619e96104bca4,d5f53948611b5c92] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:52:10.024 INFO 4486 --- [Check obproxy1,328619e96104bca4,d5f53948611b5c92] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:10.030 INFO 4486 --- [Collect obproxy2,4d2ee2c7cca99fe7,8e3c6b301a3f4951] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:52:10.030 INFO 4486 --- [Collect obproxy1,a6e943b359cc8ab4,c3a02ea8ecd36680] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:52:10.030 INFO 4486 --- [Collect obproxy1,a6e943b359cc8ab4,c3a02ea8ecd36680] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:10.030 INFO 4486 --- [Collect obproxy2,4d2ee2c7cca99fe7,8e3c6b301a3f4951] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:10.048 INFO 4486 --- [Check obproxy2,960784f256016973,c6f9ab19431d7aac] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:52:10.048 INFO 4486 --- [Check obproxy2,960784f256016973,c6f9ab19431d7aac] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:52:10.051 INFO 4486 --- [Collect obproxy2,4d2ee2c7cca99fe7,8e3c6b301a3f4951] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:52:10.055 INFO 4486 --- [Collect obproxy2,4d2ee2c7cca99fe7,8e3c6b301a3f4951] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:52:10.057 INFO 4486 --- [Check obproxy1,328619e96104bca4,d5f53948611b5c92] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:52:10.057 INFO 4486 --- [Check obproxy1,328619e96104bca4,d5f53948611b5c92] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:52:10.058 INFO 4486 --- [Collect obproxy1,a6e943b359cc8ab4,c3a02ea8ecd36680] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:52:10.060 INFO 4486 --- [Collect obproxy2,4d2ee2c7cca99fe7,8e3c6b301a3f4951] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:52:10.061 INFO 4486 --- [Collect obproxy1,a6e943b359cc8ab4,c3a02ea8ecd36680] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:52:10.064 INFO 4486 --- [Collect obproxy1,a6e943b359cc8ab4,c3a02ea8ecd36680] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:52:10.071 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,99a869d7f4c5391e,0c2d4e64677827ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=99a869d7f4c5391e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:10.071 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,99a869d7f4c5391e,0c2d4e64677827ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=99a869d7f4c5391e, duration=1 ms] 2024-10-16 13:52:10.309 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:10.959 INFO 4486 --- [ocp-schedules-18,f85c9ca8f4cf7b3e,f278a22fb059bf3a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:52:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:52:10.994 INFO 4486 --- [prometheus-minute-schedule-1,00d6e35465fbaf07,a0de82cca5545e88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:11.000 INFO 4486 --- [prometheus-minute-schedule-1,00d6e35465fbaf07,a0de82cca5545e88] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:52:11.009 INFO 4486 --- [ocp-schedules-7,2fcaa465fc2ad86a,a66a5c37098270fc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:52:11.049 WARN 4486 --- [scanner-13296383-1,2fcaa465fc2ad86a,e11cc1588557c6f2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:52:11.134 INFO 4486 --- [alarm-detect-5,2fcaa465fc2ad86a,9a7c7dde22608ee2] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057905 ~ 1729057921 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057920, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057920, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057920=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:52:11.316 INFO 4486 --- [ocp-async-5,4a686a438a44ba56,bdf05788c6d34f9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:11.605 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c15027f8a21f1357,b70a77398871a5c5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c15027f8a21f1357, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:11.605 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c15027f8a21f1357,b70a77398871a5c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c15027f8a21f1357, duration=1 ms] 2024-10-16 13:52:11.898 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ad2177124dee4d0d,7e9e2c4f34d4f9d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ad2177124dee4d0d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:11.899 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,ad2177124dee4d0d,7e9e2c4f34d4f9d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ad2177124dee4d0d, duration=2 ms] 2024-10-16 13:52:13.013 INFO 4486 --- [ocp-async-7,3b04b2420c5e2b6c,8e950a18619ae25e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:52:13.310 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:13.316 INFO 4486 --- [ocp-async-2,603c82e6a4f409cb,0f1903b241b2ca8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:13.994 INFO 4486 --- [prometheus-minute-schedule-1,379426efe22086a9,1cd1a61f75ee3a90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:14.001 INFO 4486 --- [prometheus-minute-schedule-1,379426efe22086a9,1cd1a61f75ee3a90] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:14.169 ERROR 4486 --- [metric-archive-second-20,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3866725) Duplicate entry '669741696482349056-1729057860' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:52:14.168688] [YB420AC531B8-00060B7DB03B2B15-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3866725) Duplicate entry '669741696482349056-1729057860' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:52:14.168688] [YB420AC531B8-00060B7DB03B2B15-0-0] 2024-10-16 13:52:14.174 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057860, nodeCount=1000, affectRows=1000 2024-10-16 13:52:14.178 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057860, nodeCount=998, validCount=998 2024-10-16 13:52:14.184 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057860, nodeCount=1000, affectRows=1000 2024-10-16 13:52:14.188 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057860, nodeCount=1000, validCount=1000 2024-10-16 13:52:14.204 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057860, nodeCount=1000, affectRows=1000 2024-10-16 13:52:14.209 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057860, nodeCount=1000, validCount=1000 2024-10-16 13:52:14.418 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057860, nodeCount=450, affectRows=450 2024-10-16 13:52:14.420 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057860, nodeCount=449, validCount=449 2024-10-16 13:52:15.058 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057860, nodeCount=477, affectRows=477 2024-10-16 13:52:15.317 INFO 4486 --- [ocp-async-9,6c9bf71287bc3df9,e3a3b6e73f8e906e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:15.697 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057860, nodeCount=477, validCount=477 2024-10-16 13:52:16.217 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,6819f41f1ff3a444,f45e3a8b2823b209] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=6819f41f1ff3a444, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:52:16.218 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,6819f41f1ff3a444,f45e3a8b2823b209] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=6819f41f1ff3a444, duration=2 ms] 2024-10-16 13:52:16.311 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:16.994 INFO 4486 --- [prometheus-minute-schedule-1,63f1cfa0e70d8dc2,93c1c0c0d51d325c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:17.000 INFO 4486 --- [prometheus-minute-schedule-1,63f1cfa0e70d8dc2,93c1c0c0d51d325c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:17.069 INFO 4486 --- [ocp-schedules-15,855a26e22da8b377,752a2e900f1c73fd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:52:17.210 INFO 4486 --- [ocp-schedules-30,068399e2cf1e7313,0f7b4777f3b2c106] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:52:17.214 INFO 4486 --- [ocp-schedules-20,42a0b83068585e77,c4caf6fb1e4825fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:52:17.317 INFO 4486 --- [ocp-async-5,489c0bc7bbf1e862,dcee8740dd7d4fdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:17.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,811c10d04d1fbce3,bbf90c53af9a8283] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=811c10d04d1fbce3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:17.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,811c10d04d1fbce3,bbf90c53af9a8283] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=811c10d04d1fbce3, duration=1 ms] 2024-10-16 13:52:19.133 INFO 4486 --- [ocp-schedules-29,875fc940ba2d1795,76ced4d87ca6a371] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:52:19.312 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:19.317 INFO 4486 --- [ocp-async-1,9f2980bbcf7eddcd,d47ba15cd0658aae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:19.994 INFO 4486 --- [prometheus-minute-schedule-1,622616adf441635d,a22051a26e375dad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:20.000 INFO 4486 --- [prometheus-minute-schedule-1,622616adf441635d,a22051a26e375dad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:20.076 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,a894083dddf9188d,31181f733ad58730] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a894083dddf9188d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:20.076 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,a894083dddf9188d,31181f733ad58730] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a894083dddf9188d, duration=1 ms] 2024-10-16 13:52:20.812 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:52:20.869 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 13:52:20.869 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 13:52:20.964 INFO 4486 --- [ocp-schedules-24,a7ca66413ce5f694,252ff27b8724912e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:52:20.996 INFO 4486 --- [prometheus-cache-monitor1,ef3aeba17724ecab,660f38742297a74c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:52:20.996 INFO 4486 --- [prometheus-cache-monitor1,ef3aeba17724ecab,660f38742297a74c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:52:21.224 INFO 4486 --- [ocp-schedules-23,e8ccdce84844895e,0f7a4bcfc8ef6ee2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:52:21.265 WARN 4486 --- [scanner-13297523-1,e8ccdce84844895e,7a9ce3b84446fb90] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:52:21.321 INFO 4486 --- [ocp-async-4,708907b33ec3384c,3f5de979ecc902d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:21.610 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,f3e6798c0d7d6ffe,36b12d1a1aa622fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f3e6798c0d7d6ffe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:21.610 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,f3e6798c0d7d6ffe,36b12d1a1aa622fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f3e6798c0d7d6ffe, duration=2 ms] 2024-10-16 13:52:21.903 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3c0aff6a4e5bdd99,48703faaa19c2f02] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3c0aff6a4e5bdd99, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:21.903 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3c0aff6a4e5bdd99,48703faaa19c2f02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3c0aff6a4e5bdd99, duration=1 ms] 2024-10-16 13:52:22.313 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:22.994 INFO 4486 --- [prometheus-minute-schedule-1,bd8aa8d26d5c9b82,66bd894164e8ed8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:22.999 INFO 4486 --- [prometheus-minute-schedule-1,bd8aa8d26d5c9b82,66bd894164e8ed8b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:23.014 INFO 4486 --- [ocp-async-3,d9c7a0e8910d4509,fce6773a71773ae9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:52:23.318 INFO 4486 --- [ocp-async-7,d24cee408839f85e,76dbf18374d5edfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:23.596 INFO 4486 --- [ocp-schedules-28,f1024715e5793e60,b64ce1ebd69b566d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:52:24.916 INFO 4486 --- [ocp-schedules-26,d12f8741e40e0297,719ff28ca163c375] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:52:25.313 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:25.318 INFO 4486 --- [ocp-async-8,dcc6d18325171834,57d1a5ce223bc40a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:25.994 INFO 4486 --- [prometheus-minute-schedule-1,5840df708a815ada,0f69130e89195f30] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:25.999 INFO 4486 --- [prometheus-minute-schedule-1,5840df708a815ada,0f69130e89195f30] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:26.087 INFO 4486 --- [ocp-schedules-11,202d8a8b08b69507,8c04318a43a352cc] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:52:26.087 INFO 4486 --- [ocp-schedules-11,202d8a8b08b69507,8c04318a43a352cc] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:52:26.121 INFO 4486 --- [Collect all tenants major compactions3,202d8a8b08b69507,4f3cd55f372091cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:52:26.121 INFO 4486 --- [Collect all tenants major compactions1,202d8a8b08b69507,09f5a7cfe92040d7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:52:26.121 INFO 4486 --- [Collect all tenants major compactions2,202d8a8b08b69507,d95bf5a9f4261dab] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:52:26.121 INFO 4486 --- [Collect all tenants major compactions4,202d8a8b08b69507,0018e6efc081d4db] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:52:26.121 INFO 4486 --- [Collect all tenants major compactions5,202d8a8b08b69507,1845acda7c3158c2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:52:26.134 INFO 4486 --- [Collect all tenants major compactions2,202d8a8b08b69507,d95bf5a9f4261dab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.134 INFO 4486 --- [Collect all tenants major compactions1,202d8a8b08b69507,09f5a7cfe92040d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.135 INFO 4486 --- [Collect all tenants major compactions2,202d8a8b08b69507,d95bf5a9f4261dab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:52:26.136 INFO 4486 --- [Collect all tenants major compactions1,202d8a8b08b69507,09f5a7cfe92040d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:52:26.138 INFO 4486 --- [Collect all tenants major compactions2,202d8a8b08b69507,d95bf5a9f4261dab] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:52:26.138 INFO 4486 --- [Collect all tenants major compactions1,202d8a8b08b69507,09f5a7cfe92040d7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:52:26.138 INFO 4486 --- [Collect all tenants major compactions2,202d8a8b08b69507,d95bf5a9f4261dab] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:52:26.138 INFO 4486 --- [Collect all tenants major compactions1,202d8a8b08b69507,09f5a7cfe92040d7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:52:26.140 INFO 4486 --- [Collect all tenants major compactions3,202d8a8b08b69507,4f3cd55f372091cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.142 INFO 4486 --- [Collect all tenants major compactions3,202d8a8b08b69507,4f3cd55f372091cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:52:26.145 INFO 4486 --- [Collect all tenants major compactions3,202d8a8b08b69507,4f3cd55f372091cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:52:26.145 INFO 4486 --- [Collect all tenants major compactions3,202d8a8b08b69507,4f3cd55f372091cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:52:26.153 INFO 4486 --- [Collect all tenants major compactions5,202d8a8b08b69507,1845acda7c3158c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.153 INFO 4486 --- [Collect all tenants major compactions4,202d8a8b08b69507,0018e6efc081d4db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.155 INFO 4486 --- [Collect all tenants major compactions4,202d8a8b08b69507,0018e6efc081d4db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:52:26.155 INFO 4486 --- [Collect all tenants major compactions5,202d8a8b08b69507,1845acda7c3158c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:52:26.159 INFO 4486 --- [Collect all tenants major compactions4,202d8a8b08b69507,0018e6efc081d4db] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:52:26.159 INFO 4486 --- [Collect all tenants major compactions4,202d8a8b08b69507,0018e6efc081d4db] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:52:26.160 INFO 4486 --- [Collect all tenants major compactions5,202d8a8b08b69507,1845acda7c3158c2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:52:26.160 INFO 4486 --- [Collect all tenants major compactions5,202d8a8b08b69507,1845acda7c3158c2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:52:26.163 INFO 4486 --- [ocp-schedules-11,202d8a8b08b69507,8c04318a43a352cc] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:52:26.661 INFO 4486 --- [ocp-schedules-16,b3685c4fe3a1d1d9,0ee88496a621a4d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:52:26.661 INFO 4486 --- [ocp-schedules-16,d7479e4909715b4c,7bd1790a4534b60f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:52:26.663 INFO 4486 --- [ocp-schedules-16,d7479e4909715b4c,7bd1790a4534b60f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:26.691 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.691 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:26.701 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.702 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:26.727 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:26.728 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.728 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:52:26.733 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.734 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:26.756 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:26.757 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.757 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:26.758 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:52:26.758 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.761 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:52:26.762 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:26.770 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:26.770 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:26.779 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:26.787 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:26.788 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.789 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:26.790 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:52:26.795 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:52:26.799 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:26.800 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:26.826 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:52:26.834 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:26.836 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.837 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:52:26.879 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:52:26.879 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:52:26.889 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:26.890 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.891 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:26.895 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.898 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:52:26.901 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:26.905 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:52:26.916 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:26.917 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:26.917 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:52:26.928 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:26.928 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:26.929 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:52:26.929 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:52:26.958 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:26.958 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:26.982 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:26.982 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:27.010 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:27.010 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:27.035 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:27.035 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:27.060 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:27.083 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:27.085 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.086 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:27.090 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:52:27.094 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:52:27.099 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:52:27.101 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:52:27.124 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:52:27.258 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:52:27.286 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:27.287 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.288 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:52:27.319 INFO 4486 --- [ocp-async-3,405c33c1e67b5f33,b3dacff1bcaab75a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:27.319 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:52:27.319 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:52:27.382 INFO 4486 --- [ocp-schedules-13,88fe4e6ae06ad1ec,50fbebef3a7b6402] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:52:27.384 INFO 4486 --- [ocp-schedules-13,88fe4e6ae06ad1ec,50fbebef3a7b6402] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:27.420 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:52:27.420 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.424 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:52:27.432 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.432 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:52:27.436 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:52:27.437 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.441 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:52:27.443 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.444 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:52:27.453 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.453 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:52:27.457 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.459 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:52:27.461 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.462 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:52:27.469 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.470 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.470 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:27.474 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:52:27.478 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.479 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:52:27.479 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:27.479 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.485 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.486 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.487 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:27.487 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:52:27.488 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:52:27.495 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.495 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:52:27.496 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.497 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:27.501 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.502 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:52:27.506 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.507 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:52:27.511 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.512 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.512 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.513 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.514 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.514 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.515 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.516 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.516 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.517 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.517 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.518 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:52:27.518 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.518 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.519 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.520 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.520 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.520 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.521 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:52:27.521 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.521 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.522 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.523 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.524 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.524 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.525 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.526 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.526 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.527 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:52:27.530 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:52:27.532 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.533 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:52:27.536 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.537 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:52:27.540 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.542 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.545 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.546 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.546 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.546 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:27.547 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:52:27.547 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.547 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:52:27.548 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.549 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.551 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.553 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.555 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.556 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:52:27.557 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.557 INFO 4486 --- [ocp-schedules-26,683907e8fbed3cf3,537fd05606b09f0b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:52:27.557+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:52:27.558 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.559 INFO 4486 --- [ocp-schedules-26,683907e8fbed3cf3,537fd05606b09f0b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:52:27.559 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:27.559 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.559 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:52:27.559 INFO 4486 --- [ocp-schedules-26,683907e8fbed3cf3,537fd05606b09f0b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:52:27.559 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:52:27.560 INFO 4486 --- [ocp-schedules-26,683907e8fbed3cf3,537fd05606b09f0b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:52:27.560 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.562 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.563 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.567 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.567 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.568 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:27.568 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.571 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.573 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.578 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.579 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:27.584 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:52:27.587 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:52:27.613 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.615 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.616 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:27.618 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.618 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.619 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:52:27.619 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.621 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.623 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.625 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.627 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.630 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.631 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.633 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.633 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:52:27.633 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.633 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:52:27.633 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.633 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:52:27.634 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.635 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.636 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:52:27.636 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:52:27.637 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.638 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:52:27.640 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.641 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:52:27.644 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.645 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:52:27.647 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:52:27.651 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:52:27.706 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:27.708 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:27.708 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:52:27.717 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.717 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:52:27.717 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.717 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:52:27.717 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:52:27.717 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:52:27.720 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:52:27.720 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:52:27.986 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,902131067f07b569,3722329bc0c2a20a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=902131067f07b569, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:27.986 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,902131067f07b569,3722329bc0c2a20a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=902131067f07b569, duration=1 ms] 2024-10-16 13:52:28.314 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:28.994 INFO 4486 --- [prometheus-minute-schedule-1,802d54fe4b16de1f,a74d2abe72c9c86e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:29.000 INFO 4486 --- [prometheus-minute-schedule-1,802d54fe4b16de1f,a74d2abe72c9c86e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:29.319 INFO 4486 --- [ocp-async-5,defb60dde8361f0d,0e4db6e8043dbdb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:30.080 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c32e0c6911ac8136,ac1758f5e036c481] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c32e0c6911ac8136, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:30.080 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c32e0c6911ac8136,ac1758f5e036c481] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c32e0c6911ac8136, duration=1 ms] 2024-10-16 13:52:30.103 INFO 4486 --- [prometheus-second-schedule-1,fefb60fa8bfee421,b24cacc8d097ba20] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:52:30.969 INFO 4486 --- [ocp-schedules-14,c82f2856c6822631,0ab54687f6ed0ecc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:52:31.315 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:31.319 INFO 4486 --- [ocp-async-7,11192694bc811e74,cfdffee137de3f47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:31.400 INFO 4486 --- [ocp-schedules-26,ade89fee178dbb53,1b87876e30afe671] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:52:31.422 WARN 4486 --- [scanner-13297737-1,ade89fee178dbb53,b9015b1af0c81b16] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:52:31.516 INFO 4486 --- [alarm-detect-4,ade89fee178dbb53,012785f675c00d39] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057920 ~ 1729057941 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057935, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057935, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057935=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:52:31.616 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,84ed6621bed08def,0585079dc27ec771] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=84ed6621bed08def, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:31.616 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,84ed6621bed08def,0585079dc27ec771] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=84ed6621bed08def, duration=1 ms] 2024-10-16 13:52:31.908 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,685cccbf0f719935,075af659fd5baaae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=685cccbf0f719935, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:31.908 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,685cccbf0f719935,075af659fd5baaae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=685cccbf0f719935, duration=1 ms] 2024-10-16 13:52:31.994 INFO 4486 --- [prometheus-minute-schedule-1,68cc87878d92359e,3b33b384693aae6a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:32.000 INFO 4486 --- [prometheus-minute-schedule-1,68cc87878d92359e,3b33b384693aae6a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:33.014 INFO 4486 --- [ocp-async-1,dedd2f26849e3849,d837f07dddacdd47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:52:33.319 INFO 4486 --- [ocp-async-2,65dc36baa3841610,2d814abf1864893e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:34.316 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:34.994 INFO 4486 --- [prometheus-minute-schedule-1,1c24b5b16e6e7a4c,b02bc17d366142ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:35.001 INFO 4486 --- [prometheus-minute-schedule-1,1c24b5b16e6e7a4c,b02bc17d366142ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:35.320 INFO 4486 --- [ocp-async-9,bbcd6e70d21bad5d,58ae984256d11ce0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:37.063 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:52:37.064 INFO 4486 --- [ocp-schedules-17,3c9c9441d27cc569,1ff5b6b5dcdcc10a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:52:37.066 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:52:37.066 INFO 4486 --- [ocp-schedules-25,0d55ba6d2be3aaaf,90c63a234c062d71] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:52:37.069 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:52:37.069 INFO 4486 --- [ocp-schedules-25,0d55ba6d2be3aaaf,90c63a234c062d71] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:37.073 INFO 4486 --- [ocp-schedules-17,3c9c9441d27cc569,1ff5b6b5dcdcc10a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:52:37.081 INFO 4486 --- [ocp-schedules-5,6b68f00b27560c7b,aabf50cccad935fb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:52:37.083 INFO 4486 --- [ocp-schedules-5,6b68f00b27560c7b,aabf50cccad935fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:52:37.091 INFO 4486 --- [ocp-schedules-19,1dc326986fec1b58,75b6e20a18de1b19] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:52:37.091 INFO 4486 --- [ocp-schedules-9,c163f52949b86175,79909dfe71d7411a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:52:37.091 INFO 4486 --- [ocp-schedules-20,053cd55c790c14ec,cfcc580f6d8de328] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:52:37.092 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:52:37.092 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:52:37.092 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:52:37.094 INFO 4486 --- [host-check-clockdiff2,053cd55c790c14ec,e57e021668fd9e71] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:52:37.094 INFO 4486 --- [host-check-clockdiff1,053cd55c790c14ec,5e73ff2551691c58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:52:37.094 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:52:37.094 INFO 4486 --- [host-check-clockdiff3,053cd55c790c14ec,357b9ec42b694cfa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:52:37.095 INFO 4486 --- [host-check-agent-status1,1dc326986fec1b58,fcd09924d9dc515c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:52:37.096 INFO 4486 --- [host-check-clockdiff4,053cd55c790c14ec,5c007f15e15cfe75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:52:37.096 INFO 4486 --- [host-check-agent-status2,1dc326986fec1b58,11364112478b3734] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:52:37.097 INFO 4486 --- [host-check-agent-status3,1dc326986fec1b58,33b81a06a3e7721a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:52:37.098 INFO 4486 --- [host-check-agent-status4,1dc326986fec1b58,7a883c24fcd9e633] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:52:37.099 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:37.099 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:52:37.100 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:37.101 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:52:37.102 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:52:37.103 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:52:37.103 INFO 4486 --- [host-check-agent-status1,1dc326986fec1b58,fcd09924d9dc515c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:37.113 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:37.117 INFO 4486 --- [host-check-clockdiff3,053cd55c790c14ec,357b9ec42b694cfa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057957 4 4 , error= 2024-10-16 13:52:37.117 INFO 4486 --- [host-check-clockdiff4,053cd55c790c14ec,5c007f15e15cfe75] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057957 4 4 , error= 2024-10-16 13:52:37.132 INFO 4486 --- [host-check-agent-status1,1dc326986fec1b58,fcd09924d9dc515c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:37.132 INFO 4486 --- [host-check-agent-status2,1dc326986fec1b58,11364112478b3734] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:52:37.132 INFO 4486 --- [host-check-clockdiff2,053cd55c790c14ec,e57e021668fd9e71] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057957 4 4 , error= 2024-10-16 13:52:37.146 INFO 4486 --- [host-check-clockdiff1,053cd55c790c14ec,5e73ff2551691c58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057957 0 0 , error= 2024-10-16 13:52:37.144 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:37.147 INFO 4486 --- [host-check-agent-status3,1dc326986fec1b58,33b81a06a3e7721a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:52:37.147 INFO 4486 --- [host-check-agent-status4,1dc326986fec1b58,7a883c24fcd9e633] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:37.147 INFO 4486 --- [host-check-agent-status2,1dc326986fec1b58,11364112478b3734] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:52:37.149 INFO 4486 --- [host-check-agent-status3,1dc326986fec1b58,33b81a06a3e7721a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:52:37.149 INFO 4486 --- [host-check-agent-status4,1dc326986fec1b58,7a883c24fcd9e633] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:37.151 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.152 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:52:37.169 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:52:37.170 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:52:37.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:37.172 INFO 4486 --- [host-check-agent-status4,1dc326986fec1b58,7a883c24fcd9e633] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:52:37.172 INFO 4486 --- [host-check-agent-status1,1dc326986fec1b58,fcd09924d9dc515c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:52:37.173 INFO 4486 --- [host-check-agent-status2,1dc326986fec1b58,11364112478b3734] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:52:37.173 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:52:37.172886584+08:00 2024-10-16 13:52:37.173 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:52:37.177234158+08:00 2024-10-16 13:52:37.173 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:37.174 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:52:37.174 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:52:37.173+08:00 2024-10-16 13:52:37.174 INFO 4486 --- [host-check-agent-status3,1dc326986fec1b58,33b81a06a3e7721a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:52:37.174 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:52:37.173+08:00 2024-10-16 13:52:37.175 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:52:37.176 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:52:37.176 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.177 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:52:37.180844274+08:00 2024-10-16 13:52:37.177 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:52:37.177+08:00 2024-10-16 13:52:37.177 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:52:37.173+08:00 and status AVAILABLE 2024-10-16 13:52:37.178 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:52:37.177 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:52:37.173+08:00 and status AVAILABLE 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:52:37.181670566+08:00 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-agent-status3,1dc326986fec1b58,33b81a06a3e7721a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-agent-status3,1dc326986fec1b58,33b81a06a3e7721a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-agent-status3,1dc326986fec1b58,33b81a06a3e7721a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-agent-status1,1dc326986fec1b58,fcd09924d9dc515c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-agent-status1,1dc326986fec1b58,fcd09924d9dc515c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-agent-status1,1dc326986fec1b58,fcd09924d9dc515c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:52:37.178 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:52:37.178+08:00 2024-10-16 13:52:37.179 INFO 4486 --- [host-check-agent-status4,1dc326986fec1b58,7a883c24fcd9e633] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:52:37.179 INFO 4486 --- [host-check-agent-status4,1dc326986fec1b58,7a883c24fcd9e633] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:52:37.179 INFO 4486 --- [host-check-agent-status4,1dc326986fec1b58,7a883c24fcd9e633] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:52:37.179 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.179 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.180 INFO 4486 --- [host-check-agent-status2,1dc326986fec1b58,11364112478b3734] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:52:37.180 INFO 4486 --- [host-check-agent-status2,1dc326986fec1b58,11364112478b3734] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:52:37.180 INFO 4486 --- [host-check-agent-status2,1dc326986fec1b58,11364112478b3734] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:52:37.180 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:52:37.180 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:52:37.177+08:00 and status AVAILABLE 2024-10-16 13:52:37.180 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:52:37.181 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:52:37.178+08:00 and status AVAILABLE 2024-10-16 13:52:37.181 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:52:37.182 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.183 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:52:37.183 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:52:37.188 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:52:37.190 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:52:37.190 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:37.191 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:52:37.192 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:37.192 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:37.192 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:52:37.193 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:37.194 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:52:37.195 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.195 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.195 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.195 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.195 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.196 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.196 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.197 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:37.212 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.212 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:52:37.213 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:37.222 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:37.224 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:37.224 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:37.224 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:52:37.232 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.233 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:52:37.235 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.236 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:52:37.253 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.254 INFO 4486 --- [ocp-schedules-18,3c7ec0e679f66184,00b590be2c2bee8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:52:37.256 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:52:37.257 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:52:37.266 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:52:37.267 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:52:37.270 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:52:37.281 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:52:37.282 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:37.285 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:52:37.285 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:37.294 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:52:37.301 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:37.311 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:52:37.314 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:52:37.316 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:37.318 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:52:37.320 INFO 4486 --- [ocp-async-6,f3e5fe5926c3d1d7,f5fad2ee38ed69d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:37.324 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:52:37.328 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:52:37.340 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:52:37.341 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:37.347 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:37.351 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:52:37.370 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:37.370 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:52:37.375 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:52:37.381 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:52:37.381 INFO 4486 --- [host-check-host-status3,c163f52949b86175,131a0622b40789f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:52:37.385 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:52:37.391 INFO 4486 --- [host-check-host-status4,c163f52949b86175,f0f15b0869adddc7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:52:37.401 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:52:37.410 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:52:37.422 INFO 4486 --- [host-check-host-status2,c163f52949b86175,49b24aeb81efe005] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:52:37.433 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:37.437 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.438 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:52:37.440 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.440 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.441 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:52:37.453 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.453 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.453 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.453 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.453 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.454 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.457 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:52:37.471 INFO 4486 --- [host-check-host-status1,c163f52949b86175,407328a05dd9ad65] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:52:37.482 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:52:37.530 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:52:37.579 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:37.594 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:37.596 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.597 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:52:37.598 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:52:37.599 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.600 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:52:37.601 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.602 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.603 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:52:37.614 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.614 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.614 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.614 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.614 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.615 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.622 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:52:37.838 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:52:37.846 INFO 4486 --- [ocp-schedules-8,75c6b4f2103c4ec2,87a483bb004802a7] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-10-16 13:52:37.846 INFO 4486 --- [ocp-schedules-8,75c6b4f2103c4ec2,87a483bb004802a7] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-10-16 13:52:37.881 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:37.884 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.885 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:52:37.886 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.887 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.888 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:52:37.901 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.901 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.901 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.901 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.901 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:37.902 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:37.922 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:52:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:52:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:52:37.994 INFO 4486 --- [prometheus-minute-schedule-1,b5e4b82b45c105f2,d1c3622f36543302] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:37.995 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,48ad45fbfa2c83d9,537c1ca39bc52e5e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=48ad45fbfa2c83d9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:37.995 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,48ad45fbfa2c83d9,537c1ca39bc52e5e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=48ad45fbfa2c83d9, duration=3 ms] 2024-10-16 13:52:38.001 INFO 4486 --- [prometheus-minute-schedule-1,b5e4b82b45c105f2,d1c3622f36543302] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:38.161 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:52:38.209 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:38.210 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:38.211 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:52:38.213 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:38.216 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:38.217 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:52:38.230 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:38.230 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:38.230 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:38.230 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:38.230 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:52:38.231 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:38.293 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:52:38.497 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:52:38.297684, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:52:38.497 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:52:38.297684, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:52:38.554 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:52:38.297684} } 2024-10-16 13:52:38.612 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:52:38.619 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:52:38.622 INFO 4486 --- [ocp-schedules-6,d4c7df53e61ba7e0,54526f6b1922123d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:52:39.137 INFO 4486 --- [ocp-schedules-11,85b73c0a7c51e022,dc51e5776949bc45] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:52:39.320 INFO 4486 --- [ocp-async-2,e604dcfef11df3ad,15f4178f812e3319] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:40.008 INFO 4486 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:52:40.008 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:52:40.030 INFO 4486 --- [Check obproxy2,84af542cd4b6d881,9d14b0e57e305780] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:52:40.030 INFO 4486 --- [Check obproxy2,84af542cd4b6d881,9d14b0e57e305780] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:40.030 INFO 4486 --- [Check obproxy1,16b824dd13a26bad,37c130d5321ca940] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:52:40.030 INFO 4486 --- [Check obproxy1,16b824dd13a26bad,37c130d5321ca940] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:40.035 INFO 4486 --- [Collect obproxy2,8f3658a00319da4a,0592338db257c6b7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:52:40.036 INFO 4486 --- [Collect obproxy2,8f3658a00319da4a,0592338db257c6b7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:40.036 INFO 4486 --- [Collect obproxy1,dc5a85d317c3c354,60df3c101ebb3e5d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:52:40.036 INFO 4486 --- [Collect obproxy1,dc5a85d317c3c354,60df3c101ebb3e5d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:52:40.053 INFO 4486 --- [Check obproxy2,84af542cd4b6d881,9d14b0e57e305780] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:52:40.053 INFO 4486 --- [Check obproxy2,84af542cd4b6d881,9d14b0e57e305780] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:52:40.055 INFO 4486 --- [Collect obproxy2,8f3658a00319da4a,0592338db257c6b7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:52:40.061 INFO 4486 --- [Collect obproxy2,8f3658a00319da4a,0592338db257c6b7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:52:40.061 INFO 4486 --- [Check obproxy1,16b824dd13a26bad,37c130d5321ca940] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:52:40.061 INFO 4486 --- [Check obproxy1,16b824dd13a26bad,37c130d5321ca940] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:52:40.062 INFO 4486 --- [Collect obproxy1,dc5a85d317c3c354,60df3c101ebb3e5d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:52:40.065 INFO 4486 --- [Collect obproxy1,dc5a85d317c3c354,60df3c101ebb3e5d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:52:40.066 INFO 4486 --- [Collect obproxy2,8f3658a00319da4a,0592338db257c6b7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:52:40.067 INFO 4486 --- [Collect obproxy1,dc5a85d317c3c354,60df3c101ebb3e5d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:52:40.096 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9f8543494b3fcb78,e866e1c7341ee0f2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9f8543494b3fcb78, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:40.096 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,9f8543494b3fcb78,e866e1c7341ee0f2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9f8543494b3fcb78, duration=1 ms] 2024-10-16 13:52:40.317 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:40.974 INFO 4486 --- [ocp-schedules-8,23c5f50d90cf9903,5c6fb6fe8eedb74d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:52:40.994 INFO 4486 --- [prometheus-minute-schedule-1,c33d6fc3f3abd0af,d40453beb0c66e7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:40.995 INFO 4486 --- [prometheus-hour-schedule-1,5537da4df6d49007,a797777d1dead9f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:52:41.001 INFO 4486 --- [prometheus-minute-schedule-1,c33d6fc3f3abd0af,d40453beb0c66e7a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:41.320 INFO 4486 --- [ocp-async-9,db617c508361d29b,0fef66c31c26761b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:41.587 INFO 4486 --- [ocp-schedules-12,abcce356a66a64ef,46e5b267b9b77491] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:52:41.624 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1be3912ca8ea0d14,c24884d7d590e7d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1be3912ca8ea0d14, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:41.624 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1be3912ca8ea0d14,c24884d7d590e7d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1be3912ca8ea0d14, duration=1 ms] 2024-10-16 13:52:41.630 WARN 4486 --- [scanner-13298052-1,abcce356a66a64ef,17352c133acaa8e8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:52:41.687 INFO 4486 --- [alarm-detect-8,abcce356a66a64ef,002acc329c061219] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057935 ~ 1729057951 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057950, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057950, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057950=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:52:41.912 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,62dc4265786664f0,80d4a49a4ed40086] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=62dc4265786664f0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:41.912 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,62dc4265786664f0,80d4a49a4ed40086] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=62dc4265786664f0, duration=1 ms] 2024-10-16 13:52:42.147 INFO 4486 --- [ocp-schedules-7,1aeaefca53ffc462,36b179c51ebc6521] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:52:43.014 INFO 4486 --- [ocp-async-5,85630d54b516f240,6d4fb736b67056b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:52:43.318 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:43.320 INFO 4486 --- [ocp-async-8,7c5882397ec48dcd,7f3fe052b5cd7347] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:43.600 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9a08f869ab7a1752,2db7dff5aca07894] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=9a08f869ab7a1752, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:52:43.601 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9a08f869ab7a1752,2db7dff5aca07894] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=9a08f869ab7a1752, duration=2 ms] 2024-10-16 13:52:43.994 INFO 4486 --- [prometheus-minute-schedule-1,d5949a9037424c5f,a070a0336fb6f051] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:44.002 INFO 4486 --- [prometheus-minute-schedule-1,d5949a9037424c5f,a070a0336fb6f051] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:45.321 INFO 4486 --- [ocp-async-3,d9c7e41341b9dfe0,a58ae30e44250151] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:45.546 INFO 4486 --- [ocp-schedules-27,220e1b32ae764d46,cb0302b718359e98] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:52:45.553 INFO 4486 --- [ocp-schedules-27,220e1b32ae764d46,cb0302b718359e98] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:52:45.553 INFO 4486 --- [ocp-schedules-27,220e1b32ae764d46,cb0302b718359e98] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:52:45.577 INFO 4486 --- [ocp-schedules-24,10c4ad50b149c395,a536135f7278d99f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:52:45.577 INFO 4486 --- [ocp-schedules-24,10c4ad50b149c395,a536135f7278d99f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:52:46.319 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:46.707 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d46160b56c11f559,1e38a5062ff86429] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=d46160b56c11f559, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:52:46.707 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d46160b56c11f559,1e38a5062ff86429] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=d46160b56c11f559, duration=2 ms] 2024-10-16 13:52:46.994 INFO 4486 --- [prometheus-minute-schedule-1,272eded95343dd41,0e3374b119637595] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:47.000 INFO 4486 --- [prometheus-minute-schedule-1,272eded95343dd41,0e3374b119637595] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:47.071 INFO 4486 --- [ocp-schedules-25,0d55ba6d2be3aaaf,90c63a234c062d71] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:52:47.176 INFO 4486 --- [ocp-schedules-5,6b68f00b27560c7b,aabf50cccad935fb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:52:47.213 INFO 4486 --- [ocp-schedules-20,26654c3c66d00985,fe2e64facc24c523] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:52:47.321 INFO 4486 --- [ocp-async-5,8b62585010e3153d,b45d322caaf75a5a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:48.000 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,abc3c95c2f0eaffb,240370026cf05f3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=abc3c95c2f0eaffb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:48.000 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,abc3c95c2f0eaffb,240370026cf05f3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=abc3c95c2f0eaffb, duration=1 ms] 2024-10-16 13:52:48.879 INFO 4486 --- [ocp-schedules-1,1d14edb293b9b650,3dc47b308a6e9c09] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:52:48.881 INFO 4486 --- [ocp-schedules-1,1d14edb293b9b650,3dc47b308a6e9c09] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:49.319 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:49.321 INFO 4486 --- [ocp-async-6,30d86c5073606866,c8adbfb8a204d566] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:49.994 INFO 4486 --- [prometheus-minute-schedule-1,83e11ad00ca25107,f4ced15ac72060a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:49.999 INFO 4486 --- [prometheus-minute-schedule-1,83e11ad00ca25107,f4ced15ac72060a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:50.102 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,14df6363043f79fd,64ca7cfa68a38441] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=14df6363043f79fd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:50.102 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,14df6363043f79fd,64ca7cfa68a38441] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=14df6363043f79fd, duration=1 ms] 2024-10-16 13:52:50.503 INFO 4486 --- [ocp-schedules-17,7ee5c35c7d0d011e,281ef88e24bfc7a4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:52:50.503 INFO 4486 --- [ocp-schedules-17,7ee5c35c7d0d011e,281ef88e24bfc7a4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:52:50.504 INFO 4486 --- [ocp-schedules-17,7ee5c35c7d0d011e,281ef88e24bfc7a4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:52:50.504 INFO 4486 --- [ocp-schedules-17,7ee5c35c7d0d011e,281ef88e24bfc7a4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:52:50.505 INFO 4486 --- [ocp-schedules-17,7ee5c35c7d0d011e,281ef88e24bfc7a4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:52:50.505 INFO 4486 --- [ocp-schedules-17,7ee5c35c7d0d011e,281ef88e24bfc7a4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:52:50.980 INFO 4486 --- [ocp-schedules-24,bf5061f8cfa8d190,dace3c455c21a0c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:52:51.018 INFO 4486 --- [ocp-schedules-23,acf1223637220af4,b9739a074efef571] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:52:51.018 INFO 4486 --- [ocp-schedules-23,acf1223637220af4,b9739a074efef571] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:52:51.018 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:52:51.018 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:52:51.018 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:52:51.018 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:52:51.018 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:52:51.024 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:51.024 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:52:51.024 INFO 4486 --- [ocp-schedules-23,acf1223637220af4,b9739a074efef571] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:52:51.024 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:52:51.024 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:52:51.026 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:51.026 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:52:51.026 INFO 4486 --- [ocp-schedules-23,acf1223637220af4,b9739a074efef571] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:52:51.026 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:52:51.026 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:52:51.026 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:51.028 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:51.050 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:52:51.050 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:52:51.052 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:52:51.053 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:52:51.053 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:52:51.055 INFO 4486 --- [ocp-schedules-23,acf1223637220af4,b9739a074efef571] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:52:51.073 INFO 4486 --- [ocp-schedules-23,acf1223637220af4,b9739a074efef571] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:52:51.321 INFO 4486 --- [ocp-async-2,ff3f86387f4bb63d,8c67249f1ce141ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:51.630 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6742cc1a90851ca3,3ee49e47d822f7c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6742cc1a90851ca3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:51.630 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,6742cc1a90851ca3,3ee49e47d822f7c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6742cc1a90851ca3, duration=2 ms] 2024-10-16 13:52:51.752 INFO 4486 --- [ocp-schedules-20,f4ec0cf834734745,f3142aaf1d3c1079] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:52:51.809 WARN 4486 --- [scanner-13298331-1,f4ec0cf834734745,d1c0f8e40de4c3e4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:52:51.917 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a31207295a23d13d,658ce26b232de0ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a31207295a23d13d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:51.918 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a31207295a23d13d,658ce26b232de0ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a31207295a23d13d, duration=3 ms] 2024-10-16 13:52:52.320 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:52.994 INFO 4486 --- [prometheus-minute-schedule-1,7333a75025820ebf,a6eaadde8ed2ad57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:53.003 INFO 4486 --- [prometheus-minute-schedule-1,7333a75025820ebf,a6eaadde8ed2ad57] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:53.014 INFO 4486 --- [ocp-async-4,d22f96f88fd0eab6,6a77652002feaa8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:52:53.321 INFO 4486 --- [ocp-async-9,b39388403fb31823,cba665fcb8c8c9a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:55.321 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:55.322 INFO 4486 --- [ocp-async-7,17aa23383753264d,992c9f92dca800bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:55.994 INFO 4486 --- [prometheus-minute-schedule-1,f9fd08b619a80093,7a71ffc412e8cbb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:56.001 INFO 4486 --- [prometheus-minute-schedule-1,f9fd08b619a80093,7a71ffc412e8cbb5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:57.320 INFO 4486 --- [ocp-schedules-16,d7479e4909715b4c,7bd1790a4534b60f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:52:57.320 INFO 4486 --- [ocp-schedules-16,e440b8d0759b9c41,e0cc6ee09b95a3b6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:52:57.322 INFO 4486 --- [ocp-async-3,4c99fea7dcb175da,e01a2050afd0e31c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:52:57.323 INFO 4486 --- [ocp-schedules-16,e440b8d0759b9c41,e0cc6ee09b95a3b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:52:57.353 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.353 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:57.368 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.370 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:57.390 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:57.392 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.392 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:52:57.400 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.406 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:57.428 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:57.429 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.429 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:57.430 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:52:57.431 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.434 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:52:57.435 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:57.444 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.444 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:57.454 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.465 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:57.466 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.467 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:57.467 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:52:57.472 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:52:57.477 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:52:57.479 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:52:57.505 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:52:57.521 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:57.531 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.535 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:52:57.549 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:52:57.549 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:52:57.581 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:52:57.589 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:57.590 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.591 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:52:57.591 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:57.592 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.593 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:52:57.594 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.598 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:52:57.601 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:52:57.602 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:52:57.602 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:57.628 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.628 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:57.656 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.656 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:57.682 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.682 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:57.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.711 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:52:57.735 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.736 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:52:57.766 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:52:57.796 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:57.798 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:57.799 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:52:57.803 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:52:57.808 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:52:57.813 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:52:57.815 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:52:57.840 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:52:57.863 WARN 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : Failed to get ob process info, errMsg:[AgentClient]:http request is failed, response:Cannot get process info of open /proc/1070903/status: no such file or directory, reason: %!v(MISSING) 2024-10-16 13:52:57.868 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 2024-10-16 13:52:57.873 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:52:57.875 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:52:57.898 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:52:58.006 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,52a2e36942e3794f,6eb809138be21305] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=52a2e36942e3794f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:52:58.006 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,52a2e36942e3794f,6eb809138be21305] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=52a2e36942e3794f, duration=1 ms] 2024-10-16 13:52:58.078 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000015, sqlPort: 2881 got ObProcessInfoResult(pid=91568, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:52:58.104 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:52:58.106 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:52:58.107 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:52:58.136 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:52:58.136 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:52:58.322 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:52:58.909 INFO 4486 --- [ocp-schedules-1,1d14edb293b9b650,3dc47b308a6e9c09] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:52:58.994 INFO 4486 --- [prometheus-minute-schedule-1,2f31d8323ff52c09,ce644f3cdc9dfd03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729057980, interval=60 2024-10-16 13:52:59.000 INFO 4486 --- [prometheus-minute-schedule-1,2f31d8323ff52c09,ce644f3cdc9dfd03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:52:59.142 INFO 4486 --- [ocp-schedules-21,dd18c7676869c200,c380b8297b3e35b4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:52:59.322 INFO 4486 --- [ocp-async-10,cdb13a1781584673,b838c9751a7542cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:00.069 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:53:00.087 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:00.088 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:00.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eefd893b2cbdc0c6,cafe28a158e50b61] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eefd893b2cbdc0c6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:00.107 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eefd893b2cbdc0c6,cafe28a158e50b61] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eefd893b2cbdc0c6, duration=2 ms] 2024-10-16 13:53:00.117 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:00.119 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:00.143 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:53:00.145 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:53:00.171 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:53:00.173 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:53:00.194 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:00.195 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:00.214 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:53:00.216 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:53:00.239 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:53:00.240 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:53:00.262 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:53:00.264 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:53:00.286 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:53:00.287 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:53:00.309 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:53:00.310 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:53:00.333 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:00.335 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:00.359 INFO 4486 --- [ocp-schedules-29,87f0ff031e5fb69e,2aa8c03968f5748a] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:53:00.985 INFO 4486 --- [ocp-schedules-15,d47669c38f02f942,6a0ac1348406e65b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:53:01.322 INFO 4486 --- [ocp-async-9,6fccf27589ce3ae8,920f02ae984b288a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:01.323 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:01.635 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8735de12a91069e1,ec5887c856c9ce80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8735de12a91069e1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:01.635 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8735de12a91069e1,ec5887c856c9ce80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8735de12a91069e1, duration=1 ms] 2024-10-16 13:53:01.924 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,38c5ba7327a4064b,1a131f5f4876361f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=38c5ba7327a4064b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:01.925 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,38c5ba7327a4064b,1a131f5f4876361f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=38c5ba7327a4064b, duration=2 ms] 2024-10-16 13:53:01.977 INFO 4486 --- [ocp-schedules-10,9422eabc28da9a4b,77e54f6ef370876f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:53:01.994 INFO 4486 --- [prometheus-minute-schedule-1,9da3accf69208a50,2406d795f2236bdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:02.001 INFO 4486 --- [prometheus-minute-schedule-1,9da3accf69208a50,2406d795f2236bdb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:02.018 WARN 4486 --- [scanner-13298644-1,9422eabc28da9a4b,4cc8cdd9a5287fd9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:53:02.075 INFO 4486 --- [alarm-detect-18,9422eabc28da9a4b,aca84b73c8809eb0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057950 ~ 1729057972 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057965, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057965, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057965=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:53:02.218 INFO 4486 --- [ocp-schedules-18,d05a1fb3fd80b38a,4e53cf2b4241486b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:53:02.218+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:53:02.222 INFO 4486 --- [ocp-schedules-18,d05a1fb3fd80b38a,4e53cf2b4241486b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:53:02.226 INFO 4486 --- [ocp-schedules-18,d05a1fb3fd80b38a,4e53cf2b4241486b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:53:02.240 INFO 4486 --- [ocp-schedules-18,d05a1fb3fd80b38a,4e53cf2b4241486b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:53:03.014 INFO 4486 --- [ocp-async-8,ecd6ada6feb3f203,2d70b893252be63a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:53:03.323 INFO 4486 --- [ocp-async-4,fe9613b1d8c8edc2,072fc98901a3c7a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:03.546 INFO 4486 --- [ocp-schedules-22,e32c0c76d5b3bc9b,ca6d361027681aae] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:53:03.549 INFO 4486 --- [ocp-schedules-22,e32c0c76d5b3bc9b,ca6d361027681aae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:03.550 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:53:03.550 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:53:03.550 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:53:04.324 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:04.994 INFO 4486 --- [prometheus-minute-schedule-1,0627d19d6afad16d,83923c688fbf0f13] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:05.000 INFO 4486 --- [prometheus-minute-schedule-1,0627d19d6afad16d,83923c688fbf0f13] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:05.323 INFO 4486 --- [ocp-async-5,bfc4eaa063ad3e9d,e8ddb93b5fa9975e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:07.066 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:53:07.067 INFO 4486 --- [ocp-schedules-1,381f91b07d1a3a79,f3c86cc43b97daab] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:53:07.068 INFO 4486 --- [ocp-schedules-1,381f91b07d1a3a79,f3c86cc43b97daab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:07.081 INFO 4486 --- [ocp-schedules-20,70e60cb47426b88a,42f31f300628804d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:53:07.083 INFO 4486 --- [ocp-schedules-20,70e60cb47426b88a,42f31f300628804d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:53:07.090 INFO 4486 --- [ocp-schedules-25,57075ddd752e506e,f7e7fd7eb2eccef1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:53:07.090 INFO 4486 --- [ocp-schedules-26,255f61913379868f,7af2e61a10b64156] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:53:07.090 INFO 4486 --- [ocp-schedules-27,81d45e1b098779c2,1eda16cadf305652] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:53:07.090 INFO 4486 --- [host-check-agent-status1,81d45e1b098779c2,bbf46505f5472d04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:53:07.091 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:53:07.091 INFO 4486 --- [host-check-clockdiff1,57075ddd752e506e,dc82bfb3b2b15168] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:53:07.092 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:53:07.092 INFO 4486 --- [host-check-agent-status2,81d45e1b098779c2,34782e55505e3ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:53:07.092 INFO 4486 --- [host-check-clockdiff3,57075ddd752e506e,0a014fc43e27f1ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:53:07.092 INFO 4486 --- [host-check-clockdiff2,57075ddd752e506e,37cd1d4115baab56] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:53:07.093 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:53:07.093 INFO 4486 --- [host-check-clockdiff4,57075ddd752e506e,b249e412a9f4ec3d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:53:07.096 INFO 4486 --- [host-check-agent-status3,81d45e1b098779c2,10554fcef5da2743] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:53:07.096 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:53:07.096 INFO 4486 --- [host-check-agent-status4,81d45e1b098779c2,065ebe2f22e1a8b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:53:07.121 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:07.121 INFO 4486 --- [host-check-agent-status1,81d45e1b098779c2,bbf46505f5472d04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:07.121 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:07.121 INFO 4486 --- [host-check-agent-status2,81d45e1b098779c2,34782e55505e3ad3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:07.131 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:07.131 INFO 4486 --- [host-check-agent-status3,81d45e1b098779c2,10554fcef5da2743] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:07.134 INFO 4486 --- [host-check-agent-status4,81d45e1b098779c2,065ebe2f22e1a8b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:07.134 INFO 4486 --- [host-check-agent-status1,81d45e1b098779c2,bbf46505f5472d04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:07.134 INFO 4486 --- [host-check-agent-status2,81d45e1b098779c2,34782e55505e3ad3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:07.134 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:07.134 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:07.136 INFO 4486 --- [host-check-agent-status3,81d45e1b098779c2,10554fcef5da2743] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:07.136 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:07.139 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:07.140 INFO 4486 --- [host-check-clockdiff3,57075ddd752e506e,0a014fc43e27f1ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057987 5 5 , error= 2024-10-16 13:53:07.140 INFO 4486 --- [host-check-clockdiff1,57075ddd752e506e,dc82bfb3b2b15168] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057987 0 0 , error= 2024-10-16 13:53:07.142 INFO 4486 --- [host-check-agent-status4,81d45e1b098779c2,065ebe2f22e1a8b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:07.144 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:07.154 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.155 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:07.156 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:53:07.166 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:53:07.168 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:53:07.168 INFO 4486 --- [host-check-agent-status2,81d45e1b098779c2,34782e55505e3ad3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:53:07.170 INFO 4486 --- [host-check-agent-status4,81d45e1b098779c2,065ebe2f22e1a8b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:53:07.171 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:53:07.175070841+08:00 2024-10-16 13:53:07.171 INFO 4486 --- [host-check-agent-status3,81d45e1b098779c2,10554fcef5da2743] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:53:07.172 INFO 4486 --- [host-check-agent-status1,81d45e1b098779c2,bbf46505f5472d04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:53:07.172 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:53:07.171+08:00 2024-10-16 13:53:07.172 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:53:07.176514823+08:00 2024-10-16 13:53:07.173 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.173 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:53:07.172+08:00 2024-10-16 13:53:07.174 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:07.175 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:53:07.175 INFO 4486 --- [host-check-agent-status3,81d45e1b098779c2,10554fcef5da2743] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:53:07.175 INFO 4486 --- [host-check-agent-status3,81d45e1b098779c2,10554fcef5da2743] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:53:07.175 INFO 4486 --- [host-check-agent-status3,81d45e1b098779c2,10554fcef5da2743] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:53:07.175 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:53:07.175 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:53:07.171+08:00 and status AVAILABLE 2024-10-16 13:53:07.176 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:53:07.172+08:00 and status AVAILABLE 2024-10-16 13:53:07.176 INFO 4486 --- [host-check-agent-status4,81d45e1b098779c2,065ebe2f22e1a8b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:53:07.176 INFO 4486 --- [host-check-agent-status4,81d45e1b098779c2,065ebe2f22e1a8b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:53:07.176 INFO 4486 --- [host-check-agent-status4,81d45e1b098779c2,065ebe2f22e1a8b9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-agent-status2,81d45e1b098779c2,34782e55505e3ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-agent-status2,81d45e1b098779c2,34782e55505e3ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-agent-status2,81d45e1b098779c2,34782e55505e3ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:53:07.180844649+08:00 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-agent-status1,81d45e1b098779c2,bbf46505f5472d04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-agent-status1,81d45e1b098779c2,bbf46505f5472d04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-agent-status1,81d45e1b098779c2,bbf46505f5472d04] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:53:07.177 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:53:07.177469380+08:00 2024-10-16 13:53:07.178 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:53:07.177+08:00 2024-10-16 13:53:07.178 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:53:07.177+08:00 2024-10-16 13:53:07.178 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:53:07.179 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:53:07.179 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:53:07.181 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:53:07.177+08:00 and status AVAILABLE 2024-10-16 13:53:07.181 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:53:07.177+08:00 and status AVAILABLE 2024-10-16 13:53:07.185 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.185 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:53:07.185 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:53:07.188 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:07.190 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:07.190 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:07.191 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:07.191 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:53:07.194 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:07.195 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:07.195 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:07.197 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:07.199 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.200 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:53:07.211 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.211 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:53:07.214 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:07.215 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:07.219 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:07.220 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:07.222 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.223 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:53:07.226 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.227 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:53:07.239 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:07.240 INFO 4486 --- [ocp-schedules-21,260d9b3195ebd761,788ba5d8ae9f68a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:53:07.243 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:53:07.252 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:53:07.254 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:53:07.259 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:53:07.261 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:53:07.269 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:07.271 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:53:07.274 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:53:07.279 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:07.285 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:53:07.287 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:07.300 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:07.301 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:53:07.311 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:53:07.323 INFO 4486 --- [ocp-async-8,6249b3b3950bbb2a,264b3e9c6b8723c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:07.323 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:53:07.325 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:53:07.325 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:07.328 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:07.334 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:53:07.334 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:53:07.350 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:07.351 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:07.359 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:53:07.361 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:53:07.369 INFO 4486 --- [host-check-host-status3,255f61913379868f,b18bf68af38f992b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:53:07.371 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:53:07.385 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:53:07.389 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:07.395 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:53:07.402 INFO 4486 --- [host-check-host-status4,255f61913379868f,ef8e31c530e30e96] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:53:07.406 INFO 4486 --- [host-check-host-status2,255f61913379868f,9caadf30853296b9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:53:07.451 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:53:07.460 INFO 4486 --- [host-check-host-status1,255f61913379868f,3827d6be2c873e06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:53:07.994 INFO 4486 --- [prometheus-minute-schedule-1,effdeac0f301757d,17c91a4c97b0b9f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:08.005 INFO 4486 --- [prometheus-minute-schedule-1,effdeac0f301757d,17c91a4c97b0b9f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:08.016 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f7ade988638405f0,2ece6313ded84335] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f7ade988638405f0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:08.016 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f7ade988638405f0,2ece6313ded84335] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f7ade988638405f0, duration=4 ms] 2024-10-16 13:53:08.061 INFO 4486 --- [host-check-clockdiff4,57075ddd752e506e,b249e412a9f4ec3d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057988 4 4 , error= 2024-10-16 13:53:08.902 INFO 4486 --- [host-check-clockdiff2,57075ddd752e506e,37cd1d4115baab56] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729057988 4 4 , error= 2024-10-16 13:53:09.323 INFO 4486 --- [ocp-async-7,21afd6ea05fe6577,2231fd9c5bc32004] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:10.005 INFO 4486 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:53:10.006 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:53:10.023 INFO 4486 --- [Check obproxy1,c97cfc78a3dcbe32,025c187804eb586a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:53:10.023 INFO 4486 --- [Check obproxy1,c97cfc78a3dcbe32,025c187804eb586a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:10.024 INFO 4486 --- [Check obproxy2,9a0a9e65e94a4343,fb5609206936426f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:53:10.024 INFO 4486 --- [Check obproxy2,9a0a9e65e94a4343,fb5609206936426f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:10.029 INFO 4486 --- [Collect obproxy1,b605ee8caa8369b6,2bda25112e446870] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:53:10.029 INFO 4486 --- [Collect obproxy2,f9e4c7caf87af468,825119bf6d971578] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:53:10.029 INFO 4486 --- [Collect obproxy1,b605ee8caa8369b6,2bda25112e446870] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:10.029 INFO 4486 --- [Collect obproxy2,f9e4c7caf87af468,825119bf6d971578] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:10.047 INFO 4486 --- [Check obproxy2,9a0a9e65e94a4343,fb5609206936426f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:53:10.047 INFO 4486 --- [Check obproxy2,9a0a9e65e94a4343,fb5609206936426f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:53:10.049 INFO 4486 --- [Collect obproxy2,f9e4c7caf87af468,825119bf6d971578] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:53:10.055 INFO 4486 --- [Collect obproxy2,f9e4c7caf87af468,825119bf6d971578] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:53:10.058 INFO 4486 --- [Collect obproxy1,b605ee8caa8369b6,2bda25112e446870] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:53:10.059 INFO 4486 --- [Check obproxy1,c97cfc78a3dcbe32,025c187804eb586a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:53:10.059 INFO 4486 --- [Check obproxy1,c97cfc78a3dcbe32,025c187804eb586a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:53:10.060 INFO 4486 --- [Collect obproxy2,f9e4c7caf87af468,825119bf6d971578] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:53:10.063 INFO 4486 --- [Collect obproxy1,b605ee8caa8369b6,2bda25112e446870] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:53:10.068 INFO 4486 --- [Collect obproxy1,b605ee8caa8369b6,2bda25112e446870] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:53:10.112 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1aa35ac28c09f2a1,a9fe7627ab80178e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1aa35ac28c09f2a1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:10.112 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1aa35ac28c09f2a1,a9fe7627ab80178e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1aa35ac28c09f2a1, duration=1 ms] 2024-10-16 13:53:10.326 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:53:10.991 INFO 4486 --- [ocp-schedules-9,98f8d5792a27a242,5e18e3774cb5b259] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:53:10.994 INFO 4486 --- [prometheus-minute-schedule-1,f329fce21f4f415a,01b81613bfbd4397] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:10.999 INFO 4486 --- [prometheus-minute-schedule-1,f329fce21f4f415a,01b81613bfbd4397] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:11.009 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:53:11.193 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:53:11.193 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:53:11.266 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:53:11.267 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:53:11.267 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:53:11.268 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:53:11.279 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:53:11.279 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:53:11.279 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:53:11.279 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:53:11.326 INFO 4486 --- [ocp-async-3,d99765e790216cfe,001211a262ede7dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:11.421 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:53:11.435 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:53:11.435 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:53:11.435 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:53:11.435 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:53:11.435 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:53:11.448 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:53:11.640 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,575d0c5cc06b9923,658c60686b423415] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=575d0c5cc06b9923, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:11.640 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,575d0c5cc06b9923,658c60686b423415] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=575d0c5cc06b9923, duration=0 ms] 2024-10-16 13:53:11.831 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:53:11.928 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bf3ae91b8259bc20,243b5d9eda536a3d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bf3ae91b8259bc20, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:11.928 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bf3ae91b8259bc20,243b5d9eda536a3d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bf3ae91b8259bc20, duration=1 ms] 2024-10-16 13:53:12.152 INFO 4486 --- [ocp-schedules-18,1bc5df80c2e722ba,e7a10f4d62e810d8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:53:12.186 WARN 4486 --- [scanner-13299726-1,1bc5df80c2e722ba,49999304c1fe9e46] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:53:12.251 INFO 4486 --- [alarm-detect-14,1bc5df80c2e722ba,b298fc9f8006f46a] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057965 ~ 1729057982 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057980, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057980, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057980=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:53:12.285 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.006034013S 2024-10-16 13:53:12.285 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:53:12.354 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:53:12.354 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:53:12.379 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:53:12.452 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:53:12.452 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:53:12.452 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:53:12.452 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:53:12.452 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:53:12.452 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:53:12.465 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:53:12.621 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] o.s.boot.SpringApplication : Started application in 1.355 seconds (JVM running for 78895.672) 2024-10-16 13:53:12.621 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.356135824S 2024-10-16 13:53:12.940 INFO 4486 --- [ocp-schedules-10,a0f4e3027d01ae0c,7033e740c9fa3e15] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:53:13.015 INFO 4486 --- [ocp-async-4,664c1c638b779992,f23620da09408bb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:53:13.325 INFO 4486 --- [ocp-async-9,5a65302ee88577f6,a7d71a7e8a51c9f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:13.326 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:13.388 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ac5e7cafc907d250,82814df2e368bbc2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=ac5e7cafc907d250, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:53:13.389 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ac5e7cafc907d250,82814df2e368bbc2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=ac5e7cafc907d250, duration=4 ms] 2024-10-16 13:53:13.550 INFO 4486 --- [ocp-schedules-22,e32c0c76d5b3bc9b,ca6d361027681aae] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:53:13.994 INFO 4486 --- [prometheus-minute-schedule-1,6246143410c88ca4,a5a70fb0a66caa56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:14.001 INFO 4486 --- [prometheus-minute-schedule-1,6246143410c88ca4,a5a70fb0a66caa56] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:14.197 ERROR 4486 --- [metric-archive-second-16,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '669741696503320576-1729057920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:53:14.196778] [YB420AC531B8-00060B7DB41967FD-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '669741696503320576-1729057920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:53:14.196778] [YB420AC531B8-00060B7DB41967FD-0-0] 2024-10-16 13:53:14.199 ERROR 4486 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2323956) Duplicate entry '586774769053536256-1729057920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:53:14.198548] [YB420AC531B8-00060B7DC0C8753C-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2323956) Duplicate entry '586774769053536256-1729057920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:53:14.198548] [YB420AC531B8-00060B7DC0C8753C-0-0] 2024-10-16 13:53:14.206 ERROR 4486 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=835727) Duplicate entry '586774768457945088-1729057920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:53:14.205988] [YB420AC531B8-00060B7D84EAB2A3-0-0]; nested exception is java.sql.BatchUpdateException: (conn=835727) Duplicate entry '586774768457945088-1729057920' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:53:14.205988] [YB420AC531B8-00060B7D84EAB2A3-0-0] 2024-10-16 13:53:14.224 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057920, nodeCount=1000, affectRows=1000 2024-10-16 13:53:14.228 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057920, nodeCount=998, validCount=998 2024-10-16 13:53:14.249 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057920, nodeCount=1000, affectRows=1000 2024-10-16 13:53:14.253 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057920, nodeCount=1000, validCount=1000 2024-10-16 13:53:14.366 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057920, nodeCount=265, affectRows=265 2024-10-16 13:53:14.367 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057920, nodeCount=265, validCount=265 2024-10-16 13:53:14.417 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057920, nodeCount=372, affectRows=372 2024-10-16 13:53:14.418 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057920, nodeCount=372, validCount=372 2024-10-16 13:53:14.627 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057920, nodeCount=813, affectRows=813 2024-10-16 13:53:14.630 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057920, nodeCount=812, validCount=812 2024-10-16 13:53:15.096 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057920, nodeCount=480, affectRows=480 2024-10-16 13:53:15.325 INFO 4486 --- [ocp-async-2,697be39415a56f25,6992f4d274751c9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:15.826 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057920, nodeCount=480, validCount=480 2024-10-16 13:53:16.327 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:16.994 INFO 4486 --- [prometheus-minute-schedule-1,aec6bba00aacc849,957e86fd28fad2f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:16.999 INFO 4486 --- [prometheus-minute-schedule-1,aec6bba00aacc849,957e86fd28fad2f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:17.070 INFO 4486 --- [ocp-schedules-1,381f91b07d1a3a79,f3c86cc43b97daab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:53:17.179 INFO 4486 --- [ocp-schedules-20,70e60cb47426b88a,42f31f300628804d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:53:17.216 INFO 4486 --- [ocp-schedules-14,42b6bd79b341ad06,539c895c94c3fdf2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:53:17.325 INFO 4486 --- [ocp-async-10,6e9baa9769ac9f41,0786ba52b90112c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:18.022 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,268de286fbbc50f0,ddf02777641e5025] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=268de286fbbc50f0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:18.022 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,268de286fbbc50f0,ddf02777641e5025] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=268de286fbbc50f0, duration=1 ms] 2024-10-16 13:53:19.147 INFO 4486 --- [ocp-schedules-19,943f6f5f3b63c8ec,e8c19e00f96cd95e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:53:19.325 INFO 4486 --- [ocp-async-5,896e80835300ab54,e35ff648a89460eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:19.328 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:19.994 INFO 4486 --- [prometheus-minute-schedule-1,42e6c178e3443e9d,ec80b8eacbabe477] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:20.000 INFO 4486 --- [prometheus-minute-schedule-1,42e6c178e3443e9d,ec80b8eacbabe477] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:20.119 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,212bbaaf118301e2,f07e707be0d2abe8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=212bbaaf118301e2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:20.119 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,212bbaaf118301e2,f07e707be0d2abe8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=212bbaaf118301e2, duration=2 ms] 2024-10-16 13:53:20.996 INFO 4486 --- [prometheus-cache-monitor1,9f2d26ebb6ff6161,8bad88a55005f9f1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:53:20.996 INFO 4486 --- [prometheus-cache-monitor1,9f2d26ebb6ff6161,8bad88a55005f9f1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:53:20.998 INFO 4486 --- [ocp-schedules-21,cd27189b616e0a47,a75d74aef65e3d31] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:53:21.326 INFO 4486 --- [ocp-async-9,b75c96f1586e5bce,44f933d0166b49b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:21.646 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,504760d70615d145,e61a3ca6adb8fbec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=504760d70615d145, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:21.646 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,504760d70615d145,e61a3ca6adb8fbec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=504760d70615d145, duration=1 ms] 2024-10-16 13:53:21.932 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ea9b236defcc05fd,bfd5f4fce83f3d9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ea9b236defcc05fd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:21.932 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ea9b236defcc05fd,bfd5f4fce83f3d9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ea9b236defcc05fd, duration=1 ms] 2024-10-16 13:53:22.319 INFO 4486 --- [ocp-schedules-12,058bfe12a04265e8,52db8a13a983165f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:53:22.330 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:22.344 WARN 4486 --- [scanner-13300027-1,058bfe12a04265e8,70d5a1d6b0bdecb1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:53:22.994 INFO 4486 --- [prometheus-minute-schedule-1,30d4e2ac31597db2,cbe41f0e224693dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:22.999 INFO 4486 --- [prometheus-minute-schedule-1,30d4e2ac31597db2,cbe41f0e224693dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:23.015 INFO 4486 --- [ocp-async-6,4adafb43723f1c79,e97d2e28689c0394] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:53:23.326 INFO 4486 --- [ocp-async-4,fae97cc6bc612c50,c497d82241911fb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:23.626 INFO 4486 --- [ocp-schedules-12,1284b646acaf2c04,fe6eb1a403e53f0a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:53:24.953 INFO 4486 --- [ocp-schedules-10,3fb5065856d9a74f,9d0c16c041f1eb05] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:53:25.326 INFO 4486 --- [ocp-async-1,6bcf7fcb5254f714,b49300ea5a8a5a12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:25.331 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:25.994 INFO 4486 --- [prometheus-minute-schedule-1,71922c2b904574fc,2ca2b786028c2005] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:26.000 INFO 4486 --- [prometheus-minute-schedule-1,71922c2b904574fc,2ca2b786028c2005] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:26.163 INFO 4486 --- [ocp-schedules-7,cdf942c51ac19805,a275346c27fc1531] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:53:26.163 INFO 4486 --- [ocp-schedules-7,cdf942c51ac19805,a275346c27fc1531] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:53:26.198 INFO 4486 --- [Collect all tenants major compactions1,cdf942c51ac19805,4cb6cc0245876d99] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:53:26.198 INFO 4486 --- [Collect all tenants major compactions2,cdf942c51ac19805,01c494b64ff61a73] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:53:26.198 INFO 4486 --- [Collect all tenants major compactions3,cdf942c51ac19805,27144c6eeff57a90] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:53:26.199 INFO 4486 --- [Collect all tenants major compactions5,cdf942c51ac19805,9bd7688ab21f7a80] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:53:26.199 INFO 4486 --- [Collect all tenants major compactions4,cdf942c51ac19805,034ababa4e887dfc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:53:26.216 INFO 4486 --- [Collect all tenants major compactions1,cdf942c51ac19805,4cb6cc0245876d99] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:26.216 INFO 4486 --- [Collect all tenants major compactions2,cdf942c51ac19805,01c494b64ff61a73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:26.217 INFO 4486 --- [Collect all tenants major compactions1,cdf942c51ac19805,4cb6cc0245876d99] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:53:26.218 INFO 4486 --- [Collect all tenants major compactions2,cdf942c51ac19805,01c494b64ff61a73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:53:26.220 INFO 4486 --- [Collect all tenants major compactions1,cdf942c51ac19805,4cb6cc0245876d99] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:53:26.220 INFO 4486 --- [Collect all tenants major compactions2,cdf942c51ac19805,01c494b64ff61a73] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:53:26.222 INFO 4486 --- [Collect all tenants major compactions3,cdf942c51ac19805,27144c6eeff57a90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:26.224 INFO 4486 --- [Collect all tenants major compactions3,cdf942c51ac19805,27144c6eeff57a90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:53:26.224 INFO 4486 --- [Collect all tenants major compactions1,cdf942c51ac19805,4cb6cc0245876d99] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:53:26.227 INFO 4486 --- [Collect all tenants major compactions2,cdf942c51ac19805,01c494b64ff61a73] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:53:26.228 INFO 4486 --- [Collect all tenants major compactions3,cdf942c51ac19805,27144c6eeff57a90] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:53:26.228 INFO 4486 --- [Collect all tenants major compactions3,cdf942c51ac19805,27144c6eeff57a90] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:53:26.232 INFO 4486 --- [Collect all tenants major compactions5,cdf942c51ac19805,9bd7688ab21f7a80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:26.232 INFO 4486 --- [Collect all tenants major compactions4,cdf942c51ac19805,034ababa4e887dfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:26.234 INFO 4486 --- [Collect all tenants major compactions5,cdf942c51ac19805,9bd7688ab21f7a80] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:53:26.234 INFO 4486 --- [Collect all tenants major compactions4,cdf942c51ac19805,034ababa4e887dfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:53:26.237 INFO 4486 --- [Collect all tenants major compactions4,cdf942c51ac19805,034ababa4e887dfc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:53:26.238 INFO 4486 --- [Collect all tenants major compactions4,cdf942c51ac19805,034ababa4e887dfc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:53:26.238 INFO 4486 --- [Collect all tenants major compactions5,cdf942c51ac19805,9bd7688ab21f7a80] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:53:26.239 INFO 4486 --- [Collect all tenants major compactions5,cdf942c51ac19805,9bd7688ab21f7a80] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:53:26.240 INFO 4486 --- [ocp-schedules-7,cdf942c51ac19805,a275346c27fc1531] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:53:27.326 INFO 4486 --- [ocp-async-6,644881e1036f62dd,f5ba52381596e361] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:27.386 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0376e9c16b14589b,728799b43f5c3d88] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=0376e9c16b14589b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:53:27.387 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0376e9c16b14589b,728799b43f5c3d88] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=0376e9c16b14589b, duration=2 ms] 2024-10-16 13:53:27.592 INFO 4486 --- [ocp-schedules-27,21c794de4eccc9e5,ec20f53bdb557531] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:53:27.591+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:53:27.593 INFO 4486 --- [ocp-schedules-27,21c794de4eccc9e5,ec20f53bdb557531] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:53:27.593 INFO 4486 --- [ocp-schedules-27,21c794de4eccc9e5,ec20f53bdb557531] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:53:27.594 INFO 4486 --- [ocp-schedules-27,21c794de4eccc9e5,ec20f53bdb557531] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:53:27.721 INFO 4486 --- [ocp-schedules-13,88fe4e6ae06ad1ec,50fbebef3a7b6402] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:53:28.029 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8af90d40e7b14b18,187ed615ab130b17] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8af90d40e7b14b18, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:28.029 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8af90d40e7b14b18,187ed615ab130b17] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8af90d40e7b14b18, duration=1 ms] 2024-10-16 13:53:28.093 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b5cd8a7536a1ec70,74f086ddd2ff6817] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=b5cd8a7536a1ec70, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:53:28.094 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b5cd8a7536a1ec70,74f086ddd2ff6817] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=b5cd8a7536a1ec70, duration=2 ms] 2024-10-16 13:53:28.136 INFO 4486 --- [ocp-schedules-16,e440b8d0759b9c41,e0cc6ee09b95a3b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:53:28.136 INFO 4486 --- [ocp-schedules-16,15cea0011e5523d9,0407b9e9218f72ce] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:53:28.138 INFO 4486 --- [ocp-schedules-16,15cea0011e5523d9,0407b9e9218f72ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:28.171 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.172 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:28.179 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.180 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:28.204 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.205 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:28.208 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.215 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.216 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:53:28.291 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.292 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.293 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:28.294 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:53:28.298 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.301 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:53:28.302 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:28.315 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.315 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:28.328 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.332 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:28.338 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.338 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.339 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:28.339 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:53:28.344 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:53:28.349 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:28.351 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:28.364 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.366 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.366 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:53:28.377 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:53:28.404 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:53:28.404 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:53:28.416 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.418 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.419 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:28.422 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.425 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:53:28.429 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:28.448 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:53:28.458 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.464 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.464 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:28.465 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.466 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:53:28.475 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:53:28.475 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:53:28.493 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.493 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:28.519 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.519 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:28.551 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.551 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:28.576 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.576 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:28.600 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:28.625 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.626 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.627 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:28.631 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:53:28.635 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:53:28.640 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:53:28.641 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:53:28.665 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:53:28.793 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:53:28.824 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:28.826 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:28.827 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:53:28.859 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:53:28.859 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057428392000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:53:28.994 INFO 4486 --- [prometheus-minute-schedule-1,91ebbfe49d660a81,2b4b284badcc7d56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:29.002 INFO 4486 --- [prometheus-minute-schedule-1,91ebbfe49d660a81,2b4b284badcc7d56] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:29.326 INFO 4486 --- [ocp-async-2,bc20fd2ed2a469e6,5fd2a50119ad222c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:29.665 INFO 4486 --- [ocp-schedules-2,bf0dbc2bcb3c1e56,7d87249fed337697] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-10-16 13:53:29.667 INFO 4486 --- [ocp-schedules-2,bf0dbc2bcb3c1e56,7d87249fed337697] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:29.695 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:29.697 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:29.699 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:03:03.102283, 2024-10-16 13:53:29.673, 1000] 2024-10-16 13:53:29.701 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-10-16 02:03:03.102283, endTime 2024-10-16 13:53:29.673 2024-10-16 13:53:29.702 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:42:50.879398, 2024-10-16 13:53:29.673, 1000] 2024-10-16 13:53:29.707 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:29.707 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 5, startTime 2024-10-16 08:42:50.879398, endTime 2024-10-16 13:53:29.673 2024-10-16 13:53:29.713 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:06:51.05999, 2024-10-16 13:53:29.673, 1000] 2024-10-16 13:53:29.716 INFO 4486 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:29.720 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000003, startTime 2024-10-16 02:06:51.05999, endTime 2024-10-16 13:53:29.673 2024-10-16 13:53:29.720 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:29.729 INFO 4486 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:29.730 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:29.748 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:00:14.755425, 2024-10-16 13:53:29.706, 1000] 2024-10-16 13:53:29.749 INFO 4486 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:29.749 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-10-16 08:00:14.755425, endTime 2024-10-16 13:53:29.706 2024-10-16 13:53:29.751 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:29.760 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 12:59:45.930954, 2024-10-16 13:53:29.712, 1000] 2024-10-16 13:53:29.762 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 5, startTime 2024-10-16 12:59:45.930954, endTime 2024-10-16 13:53:29.712 2024-10-16 13:53:29.780 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-12 15:26:31.336743, 2024-10-16 13:53:29.723, 1000] 2024-10-16 13:53:29.785 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000003, startTime 2024-10-12 15:26:31.336743, endTime 2024-10-16 13:53:29.723 2024-10-16 13:53:30.103 INFO 4486 --- [prometheus-second-schedule-1,5638b2144ecb2f40,eaab0949d3eae9d2] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:53:30.124 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f4166659d351d5a8,c335c8bdc51eb558] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f4166659d351d5a8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:30.124 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f4166659d351d5a8,c335c8bdc51eb558] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f4166659d351d5a8, duration=1 ms] 2024-10-16 13:53:31.004 INFO 4486 --- [ocp-schedules-6,1346ee2d63ad35ac,b7fcfa46102ea832] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:53:31.327 INFO 4486 --- [ocp-async-10,597aa866007f2ab4,1892458ee010dca4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:31.334 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:31.651 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b9adbbd3866aeb77,9827f640bddee86f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b9adbbd3866aeb77, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:31.651 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b9adbbd3866aeb77,9827f640bddee86f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b9adbbd3866aeb77, duration=1 ms] 2024-10-16 13:53:31.936 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,586002c3d5eca74b,906f1b13ccc3fe36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=586002c3d5eca74b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:31.936 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,586002c3d5eca74b,906f1b13ccc3fe36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=586002c3d5eca74b, duration=0 ms] 2024-10-16 13:53:31.994 INFO 4486 --- [prometheus-minute-schedule-1,525bd88ea0d62ac6,62c4cdbe4aaa8146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:32.000 INFO 4486 --- [prometheus-minute-schedule-1,525bd88ea0d62ac6,62c4cdbe4aaa8146] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:32.459 INFO 4486 --- [ocp-schedules-15,c9d85795665ea68c,82697ad21c0500ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:53:32.493 WARN 4486 --- [scanner-13300281-1,c9d85795665ea68c,efdda0b2c2286326] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:53:32.578 INFO 4486 --- [alarm-detect-4,c9d85795665ea68c,aadf6052222dd921] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057980 ~ 1729058002 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057995, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729057995, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729057995=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:53:33.015 INFO 4486 --- [ocp-async-8,6b76b4b25424226b,76830b09b8caeb03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:53:33.327 INFO 4486 --- [ocp-async-9,afbc92c5710dbb00,674964c1de54d748] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:34.335 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:34.994 INFO 4486 --- [prometheus-minute-schedule-1,9de0c4919157d9e8,6610ca0035208deb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:35.003 INFO 4486 --- [prometheus-minute-schedule-1,9de0c4919157d9e8,6610ca0035208deb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:35.327 INFO 4486 --- [ocp-async-3,8fe5bb6b9ddbffd8,e1ff42010d97a0a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:37.063 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:53:37.064 INFO 4486 --- [ocp-schedules-1,6d5a45f3f441673d,0b9d150ae5b9246e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:53:37.066 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:53:37.067 INFO 4486 --- [ocp-schedules-25,869953abd74bbc9a,e33451c0aa758956] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:53:37.068 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:53:37.069 INFO 4486 --- [ocp-schedules-25,869953abd74bbc9a,e33451c0aa758956] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:37.070 INFO 4486 --- [ocp-schedules-1,6d5a45f3f441673d,0b9d150ae5b9246e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:53:37.081 INFO 4486 --- [ocp-schedules-30,a7b1f6006ca4abf2,dc485428eb664c70] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:53:37.084 INFO 4486 --- [ocp-schedules-30,a7b1f6006ca4abf2,dc485428eb664c70] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:53:37.090 INFO 4486 --- [ocp-schedules-20,74e340bf36a97a4a,6441cbf2468d1309] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:53:37.091 INFO 4486 --- [ocp-schedules-4,5ce66def3e5a05eb,0df0ae08da592503] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:53:37.092 INFO 4486 --- [host-check-agent-status2,5ce66def3e5a05eb,599b5a04f79536b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:53:37.092 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:53:37.092 INFO 4486 --- [ocp-schedules-21,551dc9de3d1d1091,272eb9131ab29592] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:53:37.092 INFO 4486 --- [host-check-agent-status3,5ce66def3e5a05eb,a5ac60fa00a121ea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:53:37.092 INFO 4486 --- [host-check-agent-status1,5ce66def3e5a05eb,b23e0c1873fa569a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:53:37.092 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:53:37.093 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:53:37.093 INFO 4486 --- [host-check-agent-status4,5ce66def3e5a05eb,3a2cb5e58bd4f63d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:53:37.093 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:53:37.094 INFO 4486 --- [host-check-clockdiff1,551dc9de3d1d1091,33e8da4f27f70593] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:53:37.094 INFO 4486 --- [host-check-clockdiff2,551dc9de3d1d1091,5c7e9ba8743735a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:53:37.094 INFO 4486 --- [host-check-clockdiff3,551dc9de3d1d1091,b6aeaad631cc51d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:53:37.095 INFO 4486 --- [host-check-clockdiff4,551dc9de3d1d1091,b2f2d772b82e27fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:53:37.101 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:37.101 INFO 4486 --- [host-check-agent-status1,5ce66def3e5a05eb,b23e0c1873fa569a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:37.102 INFO 4486 --- [host-check-agent-status2,5ce66def3e5a05eb,599b5a04f79536b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:37.102 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:37.106 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:37.106 INFO 4486 --- [host-check-agent-status1,5ce66def3e5a05eb,b23e0c1873fa569a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:37.109 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:37.109 INFO 4486 --- [host-check-agent-status2,5ce66def3e5a05eb,599b5a04f79536b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:37.110 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:37.115 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:37.119 INFO 4486 --- [host-check-agent-status3,5ce66def3e5a05eb,a5ac60fa00a121ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:37.119 INFO 4486 --- [host-check-agent-status4,5ce66def3e5a05eb,3a2cb5e58bd4f63d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:37.120 INFO 4486 --- [host-check-clockdiff2,551dc9de3d1d1091,5c7e9ba8743735a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058017 5 5 , error= 2024-10-16 13:53:37.122 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:37.122 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:37.124 INFO 4486 --- [host-check-clockdiff1,551dc9de3d1d1091,33e8da4f27f70593] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058017 0 0 , error= 2024-10-16 13:53:37.127 INFO 4486 --- [host-check-agent-status3,5ce66def3e5a05eb,a5ac60fa00a121ea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:37.127 INFO 4486 --- [host-check-agent-status4,5ce66def3e5a05eb,3a2cb5e58bd4f63d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:37.130 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.131 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:37.132 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:53:37.151 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:53:37.152 INFO 4486 --- [host-check-agent-status1,5ce66def3e5a05eb,b23e0c1873fa569a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:53:37.152 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.153 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:37.154 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:53:37.155 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:53:37.158831152+08:00 2024-10-16 13:53:37.155 INFO 4486 --- [host-check-agent-status3,5ce66def3e5a05eb,a5ac60fa00a121ea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:53:37.155 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:53:37.155+08:00 2024-10-16 13:53:37.155 INFO 4486 --- [host-check-agent-status2,5ce66def3e5a05eb,599b5a04f79536b2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:53:37.156 INFO 4486 --- [host-check-agent-status4,5ce66def3e5a05eb,3a2cb5e58bd4f63d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:53:37.156 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:53:37.160643965+08:00 2024-10-16 13:53:37.156 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:53:37.157 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:37.157 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:53:37.156+08:00 2024-10-16 13:53:37.157 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:53:37.158 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:53:37.158 INFO 4486 --- [host-check-agent-status1,5ce66def3e5a05eb,b23e0c1873fa569a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:53:37.158 INFO 4486 --- [host-check-agent-status1,5ce66def3e5a05eb,b23e0c1873fa569a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:53:37.158 INFO 4486 --- [host-check-agent-status1,5ce66def3e5a05eb,b23e0c1873fa569a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:53:37.159 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:53:37.159132603+08:00 2024-10-16 13:53:37.159 INFO 4486 --- [host-check-agent-status2,5ce66def3e5a05eb,599b5a04f79536b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:53:37.159 INFO 4486 --- [host-check-agent-status2,5ce66def3e5a05eb,599b5a04f79536b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:53:37.159 INFO 4486 --- [host-check-agent-status2,5ce66def3e5a05eb,599b5a04f79536b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:53:37.159 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:53:37.163133706+08:00 2024-10-16 13:53:37.160 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:53:37.159+08:00 2024-10-16 13:53:37.160 INFO 4486 --- [host-check-agent-status3,5ce66def3e5a05eb,a5ac60fa00a121ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:53:37.160 INFO 4486 --- [host-check-agent-status3,5ce66def3e5a05eb,a5ac60fa00a121ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:53:37.160 INFO 4486 --- [host-check-agent-status3,5ce66def3e5a05eb,a5ac60fa00a121ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:53:37.160 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:53:37.159+08:00 2024-10-16 13:53:37.162 INFO 4486 --- [host-check-agent-status4,5ce66def3e5a05eb,3a2cb5e58bd4f63d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:53:37.162 INFO 4486 --- [host-check-agent-status4,5ce66def3e5a05eb,3a2cb5e58bd4f63d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:53:37.162 INFO 4486 --- [host-check-agent-status4,5ce66def3e5a05eb,3a2cb5e58bd4f63d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:53:37.165 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.168 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.169 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:53:37.169 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:53:37.155+08:00 and status AVAILABLE 2024-10-16 13:53:37.170 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:53:37.170 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.170 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.171 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:53:37.171 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:53:37.172 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:53:37.156+08:00 and status AVAILABLE 2024-10-16 13:53:37.173 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:53:37.159+08:00 and status AVAILABLE 2024-10-16 13:53:37.174 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:53:37.176 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:53:37.176 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:53:37.159+08:00 and status AVAILABLE 2024-10-16 13:53:37.179 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:53:37.181 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:37.182 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.182 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.182 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.182 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.182 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.182 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:37.183 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.184 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:37.184 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.185 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:53:37.185 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:37.185 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:37.187 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:37.189 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:37.191 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:37.195 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.196 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:53:37.205 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:53:37.207 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:37.208 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:37.209 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.210 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:53:37.211 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:37.214 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.216 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:53:37.216 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:53:37.224 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.225 INFO 4486 --- [ocp-schedules-18,4f60beff9a286b12,77bc4fe03290a66b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:53:37.238 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:53:37.239 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:53:37.245 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:53:37.250 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:53:37.250 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:53:37.255 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:53:37.273 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:53:37.279 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:37.284 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:37.285 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:37.286 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:53:37.300 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:53:37.309 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:53:37.312 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:53:37.324 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:53:37.325 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:53:37.326 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:53:37.327 INFO 4486 --- [ocp-async-5,950d49fc3676d122,599a8296b7397aff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:37.336 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:53:37.336 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:37.346 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:37.346 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:37.354 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:37.357 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:53:37.370 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:53:37.375 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:53:37.376 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:53:37.387 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:53:37.388 INFO 4486 --- [host-check-host-status3,74e340bf36a97a4a,ff4b287811a030c4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:53:37.389 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:53:37.396 INFO 4486 --- [host-check-host-status2,74e340bf36a97a4a,3b7b43ef48d602bc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:53:37.402 INFO 4486 --- [host-check-host-status4,74e340bf36a97a4a,e3cea241e154bbf3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:53:37.453 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:53:37.463 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:37.464 INFO 4486 --- [host-check-host-status1,74e340bf36a97a4a,e9dcc6006c8fc937] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:53:37.465 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.466 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:53:37.467 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.468 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.470 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:53:37.479 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.480 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.480 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.480 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.480 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.480 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.508 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:53:37.547 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:53:37.596 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:37.612 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:37.622 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.626 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:53:37.630 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:53:37.635 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.636 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:53:37.637 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.637 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.638 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:53:37.647 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.647 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.647 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.647 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.647 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.647 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.655 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:53:37.862 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:53:37.908 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:37.911 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.912 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:53:37.913 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.916 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.917 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:53:37.927 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.927 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.927 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.927 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.927 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:37.928 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:37.948 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:53:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:53:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:53:37.994 INFO 4486 --- [prometheus-minute-schedule-1,e8bd645ab40da881,59b77daefd9af673] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:38.001 INFO 4486 --- [prometheus-minute-schedule-1,e8bd645ab40da881,59b77daefd9af673] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:38.037 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,848f8685ccf4e349,36c2cc2848cd1bfa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=848f8685ccf4e349, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:38.038 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,848f8685ccf4e349,36c2cc2848cd1bfa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=848f8685ccf4e349, duration=3 ms] 2024-10-16 13:53:38.060 INFO 4486 --- [host-check-clockdiff3,551dc9de3d1d1091,b6aeaad631cc51d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058018 4 4 , error= 2024-10-16 13:53:38.071 INFO 4486 --- [host-check-clockdiff4,551dc9de3d1d1091,b2f2d772b82e27fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058018 4 4 , error= 2024-10-16 13:53:38.163 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:53:38.208 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:38.210 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:38.211 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:53:38.213 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:38.213 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:38.215 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:53:38.225 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:38.225 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:38.225 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:38.225 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:38.225 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:53:38.226 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:38.287 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:53:38.451 INFO 4486 --- [ocp-schedules-10,3f4946348c94b33a,61f1defb781239b2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:53:38.454 INFO 4486 --- [ocp-schedules-10,3f4946348c94b33a,61f1defb781239b2] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:53:38.455 INFO 4486 --- [ocp-schedules-10,3f4946348c94b33a,61f1defb781239b2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:53:38.476 INFO 4486 --- [ocp-schedules-17,33f36f97d48744f3,1698804d0b4e858c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:53:38.477 INFO 4486 --- [ocp-schedules-17,33f36f97d48744f3,1698804d0b4e858c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:53:38.501 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:53:38.291688, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:53:38.502 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:53:38.291688, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:53:38.546 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:53:38.291688} } 2024-10-16 13:53:38.613 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:53:38.621 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:53:38.624 INFO 4486 --- [ocp-schedules-13,91064dbf6f757044,a5465615e6bd3dbb] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:53:39.152 INFO 4486 --- [ocp-schedules-5,8ff6a2e1d7f484ea,d9f60cfbcf3dc7d2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:53:39.328 INFO 4486 --- [ocp-async-9,a476baf28f0d1f77,977e208e4c7c36fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:39.786 INFO 4486 --- [ocp-schedules-2,bf0dbc2bcb3c1e56,7d87249fed337697] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=3, successCount=3 2024-10-16 13:53:40.008 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:53:40.008 INFO 4486 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:53:40.028 INFO 4486 --- [Check obproxy1,10d543f576994f3c,147e28f2f326fe64] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:53:40.029 INFO 4486 --- [Check obproxy2,7876140fad467926,c06060b1ff20428a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:53:40.034 INFO 4486 --- [Check obproxy1,10d543f576994f3c,147e28f2f326fe64] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:40.035 INFO 4486 --- [Collect obproxy1,f828fb9890d5aef5,de0844f54eca6c4e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:53:40.035 INFO 4486 --- [Collect obproxy1,f828fb9890d5aef5,de0844f54eca6c4e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:40.035 INFO 4486 --- [Check obproxy2,7876140fad467926,c06060b1ff20428a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:40.036 INFO 4486 --- [Collect obproxy2,6283f131aedf379f,28681228b8f5b800] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:53:40.036 INFO 4486 --- [Collect obproxy2,6283f131aedf379f,28681228b8f5b800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:53:40.058 INFO 4486 --- [Collect obproxy2,6283f131aedf379f,28681228b8f5b800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:53:40.059 INFO 4486 --- [Check obproxy2,7876140fad467926,c06060b1ff20428a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:53:40.059 INFO 4486 --- [Check obproxy2,7876140fad467926,c06060b1ff20428a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:53:40.064 INFO 4486 --- [Collect obproxy2,6283f131aedf379f,28681228b8f5b800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:53:40.065 INFO 4486 --- [Collect obproxy1,f828fb9890d5aef5,de0844f54eca6c4e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:53:40.066 INFO 4486 --- [Check obproxy1,10d543f576994f3c,147e28f2f326fe64] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:53:40.066 INFO 4486 --- [Check obproxy1,10d543f576994f3c,147e28f2f326fe64] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:53:40.069 INFO 4486 --- [Collect obproxy2,6283f131aedf379f,28681228b8f5b800] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:53:40.070 INFO 4486 --- [Collect obproxy1,f828fb9890d5aef5,de0844f54eca6c4e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:53:40.073 INFO 4486 --- [Collect obproxy1,f828fb9890d5aef5,de0844f54eca6c4e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:53:40.130 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e9ac724a375aa991,5fa13c261722537c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e9ac724a375aa991, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:40.131 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e9ac724a375aa991,5fa13c261722537c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e9ac724a375aa991, duration=2 ms] 2024-10-16 13:53:40.337 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:40.994 INFO 4486 --- [prometheus-minute-schedule-1,a44e9f6446e08863,743d9b961cdfec4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:40.995 INFO 4486 --- [prometheus-hour-schedule-1,6b2e8c8f091b8906,7edcbad60ca911bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:53:41.001 INFO 4486 --- [prometheus-minute-schedule-1,a44e9f6446e08863,743d9b961cdfec4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:41.009 INFO 4486 --- [ocp-schedules-17,0beb3b9383f5f2dd,dfa93b8f50d3beac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:53:41.328 INFO 4486 --- [ocp-async-3,a115441f0ad8115c,adbad672df86ce51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:41.656 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d28242ce014879ff,ca5e58ba49354a4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d28242ce014879ff, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:41.657 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d28242ce014879ff,ca5e58ba49354a4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d28242ce014879ff, duration=2 ms] 2024-10-16 13:53:41.940 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bae4796191388c92,72a52ea94c6dca6e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bae4796191388c92, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:41.940 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,bae4796191388c92,72a52ea94c6dca6e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bae4796191388c92, duration=1 ms] 2024-10-16 13:53:42.155 INFO 4486 --- [ocp-schedules-29,2e094159d927b8ea,ad3669210c7f9e0c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:53:42.676 INFO 4486 --- [ocp-schedules-18,7337574e80416953,ea000868d4662434] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:53:42.727 WARN 4486 --- [scanner-13300594-1,7337574e80416953,c9da62c94b52a350] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:53:42.765 INFO 4486 --- [alarm-detect-3,7337574e80416953,0375d3ffb7c9c08a] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729057995 ~ 1729058012 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058010, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058010, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058010=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:53:43.015 INFO 4486 --- [ocp-async-2,9462f3e1a90faec9,8918bd8f786aa7a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:53:43.328 INFO 4486 --- [ocp-async-1,c5844a0c180eeae6,c48aba9a30508ca1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:43.338 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:43.994 INFO 4486 --- [prometheus-minute-schedule-1,c23f71d780bb79f7,be7d6250bfedab45] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:44.001 INFO 4486 --- [prometheus-minute-schedule-1,c23f71d780bb79f7,be7d6250bfedab45] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:45.328 INFO 4486 --- [ocp-async-7,cf5fa421899802b7,f65de4b1f70d75e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:45.554 INFO 4486 --- [ocp-schedules-27,916c2d9832fca74f,2034619920ac22fa] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:53:45.558 INFO 4486 --- [ocp-schedules-27,916c2d9832fca74f,2034619920ac22fa] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:53:45.558 INFO 4486 --- [ocp-schedules-27,916c2d9832fca74f,2034619920ac22fa] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:53:45.583 INFO 4486 --- [ocp-schedules-15,d2cd505b88e92601,b9a093e0dd96c6b0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:53:45.583 INFO 4486 --- [ocp-schedules-15,d2cd505b88e92601,b9a093e0dd96c6b0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:53:46.339 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:46.994 INFO 4486 --- [prometheus-minute-schedule-1,97ebf7fa5cd919e1,434d429620c881dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:47.000 INFO 4486 --- [prometheus-minute-schedule-1,97ebf7fa5cd919e1,434d429620c881dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:47.071 INFO 4486 --- [ocp-schedules-25,869953abd74bbc9a,e33451c0aa758956] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:53:47.158 INFO 4486 --- [ocp-schedules-30,a7b1f6006ca4abf2,dc485428eb664c70] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:53:47.219 INFO 4486 --- [ocp-schedules-26,398295573ec8e011,1ec9dd50b261d357] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:53:47.328 INFO 4486 --- [ocp-async-2,a5facc55178583a8,3b77c0fd3d846e86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:48.043 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d509ed19605377cf,446e39cbfb0036f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d509ed19605377cf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:48.043 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d509ed19605377cf,446e39cbfb0036f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d509ed19605377cf, duration=2 ms] 2024-10-16 13:53:48.779 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:48.780 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:53:48.781 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:48.782 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:53:48.782 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:53:49.329 INFO 4486 --- [ocp-async-5,cd6930d7e06503fc,0fd40252977011af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:49.340 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:49.994 INFO 4486 --- [prometheus-minute-schedule-1,2d074c50768bc136,477298a00e4ebad5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:50.000 INFO 4486 --- [prometheus-minute-schedule-1,2d074c50768bc136,477298a00e4ebad5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:50.135 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,bba53ab905a19f9e,45c72969e015cd2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bba53ab905a19f9e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:50.135 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,bba53ab905a19f9e,45c72969e015cd2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bba53ab905a19f9e, duration=1 ms] 2024-10-16 13:53:50.506 INFO 4486 --- [ocp-schedules-2,62c6fcd857387f93,9363f27b7b4dc254] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:53:50.506 INFO 4486 --- [ocp-schedules-2,62c6fcd857387f93,9363f27b7b4dc254] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:53:50.512 INFO 4486 --- [ocp-schedules-2,62c6fcd857387f93,9363f27b7b4dc254] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:53:50.512 INFO 4486 --- [ocp-schedules-2,62c6fcd857387f93,9363f27b7b4dc254] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:53:50.515 INFO 4486 --- [ocp-schedules-2,62c6fcd857387f93,9363f27b7b4dc254] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:53:50.515 INFO 4486 --- [ocp-schedules-2,62c6fcd857387f93,9363f27b7b4dc254] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:53:51.016 INFO 4486 --- [ocp-schedules-13,f29c60e0cd4f01a7,cdf4a06a23dc5196] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:53:51.080 INFO 4486 --- [ocp-schedules-15,a17928e61adf8823,cac4ab15967a8e18] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:53:51.080 INFO 4486 --- [ocp-schedules-15,a17928e61adf8823,cac4ab15967a8e18] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:53:51.080 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:53:51.080 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:53:51.080 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:53:51.080 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:53:51.080 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:53:51.085 INFO 4486 --- [ocp-schedules-15,a17928e61adf8823,cac4ab15967a8e18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:53:51.085 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:51.085 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:53:51.086 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:53:51.087 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:53:51.087 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:53:51.087 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:51.087 INFO 4486 --- [ocp-schedules-15,a17928e61adf8823,cac4ab15967a8e18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:53:51.087 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:53:51.088 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:53:51.088 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:53:51.089 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:53:51.109 INFO 4486 --- [ocp-schedules-15,a17928e61adf8823,cac4ab15967a8e18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:53:51.112 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:53:51.112 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:53:51.114 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:53:51.114 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:53:51.114 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:53:51.117 INFO 4486 --- [ocp-schedules-15,a17928e61adf8823,cac4ab15967a8e18] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:53:51.329 INFO 4486 --- [ocp-async-1,db6d53083f04ad75,e267be4e4afe7dc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:51.663 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,967575fe6613c298,992a4aad1e0ae5ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=967575fe6613c298, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:51.663 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,967575fe6613c298,992a4aad1e0ae5ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=967575fe6613c298, duration=1 ms] 2024-10-16 13:53:51.944 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,107511fe48e27691,6d57d102bdca3abf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=107511fe48e27691, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:51.944 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,107511fe48e27691,6d57d102bdca3abf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=107511fe48e27691, duration=1 ms] 2024-10-16 13:53:52.341 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:52.856 INFO 4486 --- [ocp-schedules-29,cb3b5446269d8230,e5d2891e050f3f95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:53:52.877 WARN 4486 --- [scanner-13300810-1,cb3b5446269d8230,c55cd08cd34aa841] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:53:52.995 INFO 4486 --- [prometheus-minute-schedule-1,19a1699b743ad0ad,f5b470f38e35569e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:53.014 INFO 4486 --- [prometheus-minute-schedule-1,19a1699b743ad0ad,f5b470f38e35569e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:53.019 INFO 4486 --- [ocp-async-7,555cd3494d2d4c06,1b039bdc2404b1bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:53:53.329 INFO 4486 --- [ocp-async-3,b03524eef791ba13,2a120ba2bbc49a78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:54.681 INFO 4486 --- [ocp-schedules-10,507bc49b4dc947b2,0c9917b375df5cbc] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:53:54.681+08:00, tenantSize=5 2024-10-16 13:53:54.686 INFO 4486 --- [ocp-schedules-10,507bc49b4dc947b2,0c9917b375df5cbc] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729058034681000 2024-10-16 13:53:54.686 INFO 4486 --- [ocp-schedules-10,507bc49b4dc947b2,0c9917b375df5cbc] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729057734641000 2024-10-16 13:53:54.711 INFO 4486 --- [ocp-schedules-10,507bc49b4dc947b2,0c9917b375df5cbc] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:53:54.717 INFO 4486 --- [ocp-schedules-10,507bc49b4dc947b2,0c9917b375df5cbc] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729058034714000 2024-10-16 13:53:54.718 INFO 4486 --- [ocp-schedules-10,507bc49b4dc947b2,0c9917b375df5cbc] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729057734614000 2024-10-16 13:53:54.734 INFO 4486 --- [ocp-schedules-10,507bc49b4dc947b2,0c9917b375df5cbc] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:53:55.329 INFO 4486 --- [ocp-async-8,981ac4d433a53368,10335e281b604ffb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:55.341 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:55.994 INFO 4486 --- [prometheus-minute-schedule-1,effa4eb7f7e74d57,1ceed4be04790618] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:56.001 INFO 4486 --- [prometheus-minute-schedule-1,effa4eb7f7e74d57,1ceed4be04790618] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:57.330 INFO 4486 --- [ocp-async-7,3ce9bf37b53291a5,76f4cbf10ec5ecaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:58.049 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,4fecea715fb7333f,5f696ac800251bb8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4fecea715fb7333f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:53:58.050 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,4fecea715fb7333f,5f696ac800251bb8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4fecea715fb7333f, duration=3 ms] 2024-10-16 13:53:58.343 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:53:58.859 INFO 4486 --- [ocp-schedules-16,15cea0011e5523d9,0407b9e9218f72ce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:53:58.859 INFO 4486 --- [ocp-schedules-16,4ae2d8b025a48603,1cf7a1d1cd478110] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:53:58.862 INFO 4486 --- [ocp-schedules-16,4ae2d8b025a48603,1cf7a1d1cd478110] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:53:58.889 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:58.890 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:58.900 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:58.905 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:58.923 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:58.924 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:58.924 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:58.924 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:53:58.925 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:58.952 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:58.953 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:58.953 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:58.954 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:53:58.954 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:58.957 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:53:58.958 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:58.968 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:58.968 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:58.981 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:58.992 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:58.993 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:58.994 INFO 4486 --- [prometheus-minute-schedule-1,8b6143ab9d615ff4,93a34c1b4aa5737b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058040, interval=60 2024-10-16 13:53:58.994 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:58.995 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:53:59.000 INFO 4486 --- [prometheus-minute-schedule-1,8b6143ab9d615ff4,93a34c1b4aa5737b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:53:59.003 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:53:59.010 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:53:59.013 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:53:59.043 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:59.044 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:53:59.045 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:59.046 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:53:59.101 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:53:59.101 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:53:59.107 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:59.109 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:59.110 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:53:59.113 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:59.117 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:53:59.120 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:59.138 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:53:59.148 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:59.150 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:59.151 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:53:59.154 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:59.154 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:59.156 INFO 4486 --- [ocp-schedules-18,a7d4167fc2cfa7c8,f94948c8bd3f114e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:53:59.165 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:53:59.165 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:53:59.184 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:59.184 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:59.211 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:59.211 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:59.240 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:59.240 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:53:59.266 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:59.266 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:53:59.292 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:53:59.319 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:59.320 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:59.321 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:53:59.326 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:53:59.330 INFO 4486 --- [ocp-async-4,4317cce8e85f9aec,1ecd561275661058] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:53:59.331 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:53:59.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:53:59.338 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:53:59.363 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:53:59.511 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:53:59.541 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:53:59.542 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:53:59.544 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:53:59.574 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:53:59.574 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:54:00.138 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,976205792b33eae2,83c1cb2b555ca97c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=976205792b33eae2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:00.139 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,976205792b33eae2,83c1cb2b555ca97c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=976205792b33eae2, duration=1 ms] 2024-10-16 13:54:01.021 INFO 4486 --- [ocp-schedules-4,d571acb090f93dde,6eec3c17b7f2fdd4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:54:01.330 INFO 4486 --- [ocp-async-5,099c7c0ccf3db3a1,963cf29432596825] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:01.344 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:01.668 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dd9033dc9ca0f556,f35b176bf9cb55c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dd9033dc9ca0f556, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:01.668 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dd9033dc9ca0f556,f35b176bf9cb55c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dd9033dc9ca0f556, duration=1 ms] 2024-10-16 13:54:01.956 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,989e0873764856f0,0e637d13765c323e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=989e0873764856f0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:01.956 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,989e0873764856f0,0e637d13765c323e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=989e0873764856f0, duration=1 ms] 2024-10-16 13:54:01.994 INFO 4486 --- [prometheus-minute-schedule-1,351958a5e3197589,da1763205c63a1c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:02.000 INFO 4486 --- [prometheus-minute-schedule-1,351958a5e3197589,da1763205c63a1c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:02.270 INFO 4486 --- [ocp-schedules-24,5f91c6e9fb0b53d7,e9fd760c6204ac13] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:54:02.270+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:54:02.274 INFO 4486 --- [ocp-schedules-24,5f91c6e9fb0b53d7,e9fd760c6204ac13] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:54:02.278 INFO 4486 --- [ocp-schedules-24,5f91c6e9fb0b53d7,e9fd760c6204ac13] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:54:02.290 INFO 4486 --- [ocp-schedules-24,5f91c6e9fb0b53d7,e9fd760c6204ac13] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:54:02.375 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:54:02.375 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2024-10-16 13:54:02.375 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=7 2024-10-16 13:54:02.375 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2024-10-16 13:54:02.375 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1000015 2024-10-16 13:54:02.393 WARN 4486 --- [ForkJoinPool.commonPool-worker-1,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:54:02.397 WARN 4486 --- [ForkJoinPool.commonPool-worker-13,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:54:02.410 WARN 4486 --- [ForkJoinPool.commonPool-worker-19,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.439 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 13:54:02.440 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 7 agent sql collect property 2024-10-16 13:54:02.440 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:02.443 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:02.458 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:54:02.458 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:54:02.459 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 13:54:02.459 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2024-10-16 13:54:02.462 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2024-10-16 13:54:02.469 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2024-10-16 13:54:02.472 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:02.473 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1000015 agent sql collect property 2024-10-16 13:54:02.474 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:02.474 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:02.476 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2024-10-16 13:54:02.477 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:02.481 INFO 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 7, do nothing 2024-10-16 13:54:02.486 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:02.489 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:02.506 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313637), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.184), (ob.run.path=/root/oceanbase), (ob.install.path=/root/oceanbase), (ob.data.path=/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=OceanBase), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local ser 2024-10-16 13:54:02.515 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313639), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.178), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data), (monagent.cluster.id=5), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=obce_3zones), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=2), (monagent.obproxy.cluster.name=obce_3zones), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=10.197.49.178), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLev 2024-10-16 13:54:02.518 ERROR 4486 --- [ocp-schedules-5,9265c747aeda456b,2ce01f00f1b45a65] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor1649.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:Unexpected error: key monagent.ob.cgroup.status is not found in config properties! at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:476) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:286) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:114) at com.oceanbase.ocp.executor.executor.AgentExecutor.updateAgentConfig(AgentExecutor.java:193) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentProperties(HostAgentConfigManagerImpl.java:188) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentConfig(HostAgentConfigManagerImpl.java:196) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$FastClassBySpringCGLIB$$798eb4cd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$EnhancerBySpringCGLIB$$eb50b66a.refreshAgentConfig() at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.refreshHostAgentConfig(HostAgentServiceImpl.java:734) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$5f28aa5.refreshHostAgentConfig() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$refreshAgentConfig$0(HostSchedules.java:92) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:54:02.523 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313644), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=172.28.46.152), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1000003), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=glob_3zone), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=1000002), (monagent.obproxy.cluster.name=xc_OB43), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=172.28.46.152), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition fro 2024-10-16 13:54:02.526 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2024-10-16 13:54:02.532 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1000015, do nothing 2024-10-16 13:54:03.018 INFO 4486 --- [ocp-async-10,6fd6897e672e96c7,cbf866e5d622101c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:54:03.123 INFO 4486 --- [ocp-schedules-28,77b4f409aba54f5f,e26986939e974950] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:54:03.151 WARN 4486 --- [scanner-13301189-1,77b4f409aba54f5f,20329a094bab61f8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:54:03.202 INFO 4486 --- [alarm-detect-10,77b4f409aba54f5f,6375c5a7281c2657] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058010 ~ 1729058033 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058025, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058025, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058025=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:54:03.204 INFO 4486 --- [alarm-detect-10,77b4f409aba54f5f,6375c5a7281c2657] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:54:03.330 INFO 4486 --- [ocp-async-7,d6f6ab76d8a55a15,cb27b4a704463b7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:04.345 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:04.994 INFO 4486 --- [prometheus-minute-schedule-1,03095c7c303ec5df,995fdc0fa3f46f21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:05.002 INFO 4486 --- [prometheus-minute-schedule-1,03095c7c303ec5df,995fdc0fa3f46f21] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:05.330 INFO 4486 --- [ocp-async-4,2e83df4fccda3c3d,ea42934e572ba5cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:07.066 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:54:07.067 INFO 4486 --- [ocp-schedules-26,0a61ea813d152945,3236d3304ebd438c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:54:07.068 INFO 4486 --- [ocp-schedules-26,0a61ea813d152945,3236d3304ebd438c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:07.081 INFO 4486 --- [ocp-schedules-13,34b5bb6bb973399f,17be046555bb1bbe] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:54:07.083 INFO 4486 --- [ocp-schedules-13,34b5bb6bb973399f,17be046555bb1bbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:54:07.089 INFO 4486 --- [ocp-schedules-3,c2522b47048a0003,7188fbc007cac454] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:54:07.090 INFO 4486 --- [host-check-agent-status3,c2522b47048a0003,3336d9bb42f39133] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:54:07.090 INFO 4486 --- [host-check-agent-status1,c2522b47048a0003,73cb0a96ddcdb41d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:54:07.090 INFO 4486 --- [ocp-schedules-5,0c4fb232cea638e6,e29bf1c91f8aacdb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:54:07.090 INFO 4486 --- [host-check-agent-status2,c2522b47048a0003,fc0e92a3969e387a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:54:07.091 INFO 4486 --- [host-check-agent-status4,c2522b47048a0003,f83a6ad20766a268] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:54:07.091 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:54:07.091 INFO 4486 --- [ocp-schedules-20,af4f92630bf5eb1a,25fffccad82ac1d6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:54:07.091 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:54:07.091 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:54:07.091 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:54:07.092 INFO 4486 --- [host-check-clockdiff1,af4f92630bf5eb1a,ca62d33e81b39fe8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:54:07.092 INFO 4486 --- [host-check-clockdiff2,af4f92630bf5eb1a,890958c075f0cf4f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:54:07.092 INFO 4486 --- [host-check-clockdiff3,af4f92630bf5eb1a,1b6449bee0358b8a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:54:07.092 INFO 4486 --- [host-check-clockdiff4,af4f92630bf5eb1a,cd642503a7e2c238] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:54:07.095 INFO 4486 --- [host-check-agent-status1,c2522b47048a0003,73cb0a96ddcdb41d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:07.096 INFO 4486 --- [host-check-agent-status3,c2522b47048a0003,3336d9bb42f39133] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:07.113 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:07.113 INFO 4486 --- [host-check-agent-status1,c2522b47048a0003,73cb0a96ddcdb41d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:07.114 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:07.114 INFO 4486 --- [host-check-agent-status3,c2522b47048a0003,3336d9bb42f39133] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:07.114 INFO 4486 --- [host-check-clockdiff1,af4f92630bf5eb1a,ca62d33e81b39fe8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058047 0 0 , error= 2024-10-16 13:54:07.118 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:07.118 INFO 4486 --- [host-check-clockdiff4,af4f92630bf5eb1a,cd642503a7e2c238] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058047 4 4 , error= 2024-10-16 13:54:07.118 INFO 4486 --- [host-check-clockdiff2,af4f92630bf5eb1a,890958c075f0cf4f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058047 5 5 , error= 2024-10-16 13:54:07.121 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:07.122 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:07.122 INFO 4486 --- [host-check-agent-status4,c2522b47048a0003,f83a6ad20766a268] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:07.122 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:07.122 INFO 4486 --- [host-check-agent-status2,c2522b47048a0003,fc0e92a3969e387a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:07.123 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:07.123 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:07.123 INFO 4486 --- [host-check-agent-status4,c2522b47048a0003,f83a6ad20766a268] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:07.124 INFO 4486 --- [host-check-agent-status2,c2522b47048a0003,fc0e92a3969e387a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:07.128 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.129 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:54:07.130 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:54:07.146 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:54:07.146 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:54:07.147 INFO 4486 --- [host-check-agent-status1,c2522b47048a0003,73cb0a96ddcdb41d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:54:07.148 INFO 4486 --- [host-check-agent-status2,c2522b47048a0003,fc0e92a3969e387a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:54:07.148 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.148 INFO 4486 --- [host-check-agent-status3,c2522b47048a0003,3336d9bb42f39133] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:54:07.149 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:54:07.149 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:54:07.148791976+08:00 2024-10-16 13:54:07.149 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:54:07.152997061+08:00 2024-10-16 13:54:07.149 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:54:07.150 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:54:07.149+08:00 2024-10-16 13:54:07.150 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:54:07.149+08:00 2024-10-16 13:54:07.151 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:54:07.155713377+08:00 2024-10-16 13:54:07.152 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:54:07.149+08:00 and status AVAILABLE 2024-10-16 13:54:07.152 INFO 4486 --- [host-check-agent-status4,c2522b47048a0003,f83a6ad20766a268] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:54:07.149+08:00 and status AVAILABLE 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:54:07.151+08:00 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-agent-status3,c2522b47048a0003,3336d9bb42f39133] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-agent-status3,c2522b47048a0003,3336d9bb42f39133] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-agent-status3,c2522b47048a0003,3336d9bb42f39133] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-agent-status2,c2522b47048a0003,fc0e92a3969e387a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-agent-status2,c2522b47048a0003,fc0e92a3969e387a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:54:07.153 INFO 4486 --- [host-check-agent-status2,c2522b47048a0003,fc0e92a3969e387a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:54:07.154 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:54:07.155 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:54:07.155 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:54:07.156 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:54:07.151+08:00 and status AVAILABLE 2024-10-16 13:54:07.157 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:54:07.157 INFO 4486 --- [host-check-agent-status4,c2522b47048a0003,f83a6ad20766a268] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:54:07.157 INFO 4486 --- [host-check-agent-status1,c2522b47048a0003,73cb0a96ddcdb41d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:54:07.157 INFO 4486 --- [host-check-agent-status4,c2522b47048a0003,f83a6ad20766a268] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:54:07.157 INFO 4486 --- [host-check-agent-status1,c2522b47048a0003,73cb0a96ddcdb41d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:54:07.157 INFO 4486 --- [host-check-agent-status4,c2522b47048a0003,f83a6ad20766a268] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:54:07.157 INFO 4486 --- [host-check-agent-status1,c2522b47048a0003,73cb0a96ddcdb41d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:54:07.158 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:54:07.162677495+08:00 2024-10-16 13:54:07.158 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:54:07.159 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:54:07.158+08:00 2024-10-16 13:54:07.162 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:54:07.158+08:00 and status AVAILABLE 2024-10-16 13:54:07.162 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.164 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:54:07.164 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:07.164 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:07.166 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:07.166 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:07.166 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:54:07.167 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:07.169 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:07.173 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:07.175 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:07.176 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.177 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:07.186 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:07.189 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:07.191 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:07.195 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.196 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:07.202 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:07.206 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.207 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:07.210 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.213 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:54:07.220 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:54:07.221 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:54:07.227 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:07.228 INFO 4486 --- [ocp-schedules-24,40f55aa1e8345cfa,5a796dea288c7dba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:54:07.235 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:54:07.237 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:54:07.237 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:54:07.243 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:54:07.249 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:54:07.252 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:07.254 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:07.254 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:54:07.263 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:07.269 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:07.283 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:54:07.291 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:54:07.292 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:54:07.293 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:54:07.302 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:54:07.304 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:54:07.308 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:07.320 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:07.321 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:54:07.323 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:07.331 INFO 4486 --- [ocp-async-5,47c49bc19524b6ed,86919bc83d401b20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:07.332 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:54:07.337 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:54:07.346 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:07.350 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:54:07.351 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:54:07.351 INFO 4486 --- [host-check-host-status3,0c4fb232cea638e6,6d471bfac627513e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:54:07.352 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:07.360 INFO 4486 --- [host-check-host-status4,0c4fb232cea638e6,ab138e94e54570ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:54:07.360 INFO 4486 --- [host-check-host-status2,0c4fb232cea638e6,502ce69dad645850] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:54:07.411 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:54:07.423 INFO 4486 --- [host-check-host-status1,0c4fb232cea638e6,5fbb5a2e87d5fe78] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:54:07.994 INFO 4486 --- [prometheus-minute-schedule-1,0c31a826e0b5305c,26501f77b1456a91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:07.999 INFO 4486 --- [prometheus-minute-schedule-1,0c31a826e0b5305c,26501f77b1456a91] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:08.053 INFO 4486 --- [host-check-clockdiff3,af4f92630bf5eb1a,1b6449bee0358b8a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058048 4 4 , error= 2024-10-16 13:54:08.057 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2703d0aff88564b9,b1c6b42777bb840d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2703d0aff88564b9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:08.057 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,2703d0aff88564b9,b1c6b42777bb840d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2703d0aff88564b9, duration=1 ms] 2024-10-16 13:54:09.331 INFO 4486 --- [ocp-async-6,0dbd88a52d49c20d,2c35268e782aa526] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:09.481 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0258759e813ccea8,610d288269bf70b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=0258759e813ccea8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:54:09.482 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0258759e813ccea8,610d288269bf70b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=0258759e813ccea8, duration=2 ms] 2024-10-16 13:54:10.007 INFO 4486 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:54:10.009 INFO 4486 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:54:10.031 INFO 4486 --- [Check obproxy2,15ccd81ef933f279,e24d40a5d53cd277] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:54:10.031 INFO 4486 --- [Check obproxy1,d6ea722ac0d8a36c,047eefd356497281] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:54:10.031 INFO 4486 --- [Check obproxy2,15ccd81ef933f279,e24d40a5d53cd277] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:10.031 INFO 4486 --- [Check obproxy1,d6ea722ac0d8a36c,047eefd356497281] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:10.039 INFO 4486 --- [Collect obproxy2,5102134268f488a3,820a758a9cba0745] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:54:10.039 INFO 4486 --- [Collect obproxy2,5102134268f488a3,820a758a9cba0745] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:10.040 INFO 4486 --- [Collect obproxy1,e8844dfab6bee952,4db28675f4586811] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:54:10.041 INFO 4486 --- [Collect obproxy1,e8844dfab6bee952,4db28675f4586811] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:10.060 INFO 4486 --- [Check obproxy2,15ccd81ef933f279,e24d40a5d53cd277] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:54:10.060 INFO 4486 --- [Check obproxy2,15ccd81ef933f279,e24d40a5d53cd277] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:54:10.062 INFO 4486 --- [Collect obproxy2,5102134268f488a3,820a758a9cba0745] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:54:10.063 INFO 4486 --- [Check obproxy1,d6ea722ac0d8a36c,047eefd356497281] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:54:10.063 INFO 4486 --- [Check obproxy1,d6ea722ac0d8a36c,047eefd356497281] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:54:10.068 INFO 4486 --- [Collect obproxy2,5102134268f488a3,820a758a9cba0745] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:54:10.072 INFO 4486 --- [Collect obproxy1,e8844dfab6bee952,4db28675f4586811] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:54:10.073 INFO 4486 --- [Collect obproxy2,5102134268f488a3,820a758a9cba0745] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:54:10.078 INFO 4486 --- [Collect obproxy1,e8844dfab6bee952,4db28675f4586811] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:54:10.084 INFO 4486 --- [Collect obproxy1,e8844dfab6bee952,4db28675f4586811] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:54:10.143 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bdfc0104a49dbbab,d4e0809888491a4e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bdfc0104a49dbbab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:10.143 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,bdfc0104a49dbbab,d4e0809888491a4e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bdfc0104a49dbbab, duration=1 ms] 2024-10-16 13:54:10.347 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:10.554 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c7b9cff554d205e6,e88193d9736b61d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=c7b9cff554d205e6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:54:10.555 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c7b9cff554d205e6,e88193d9736b61d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=c7b9cff554d205e6, duration=1 ms] 2024-10-16 13:54:10.984 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:54:10.994 INFO 4486 --- [prometheus-minute-schedule-1,722d50dab50af1d3,ff6a4eaf531428cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:11.001 INFO 4486 --- [prometheus-minute-schedule-1,722d50dab50af1d3,ff6a4eaf531428cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:54:11.026 INFO 4486 --- [ocp-schedules-14,7beeec8aeca37d90,4fc93d5d0b6c69d4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:54:11.151 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,77d43fdd27ac3bcb,e908e3c84807d0a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=77d43fdd27ac3bcb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:54:11.152 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,77d43fdd27ac3bcb,e908e3c84807d0a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=77d43fdd27ac3bcb, duration=2 ms] 2024-10-16 13:54:11.331 INFO 4486 --- [ocp-async-2,358e7330943b469a,4aa8ded7b953f0f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:11.680 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6c477fa07e449955,698fd5e8320524e1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6c477fa07e449955, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:11.681 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6c477fa07e449955,698fd5e8320524e1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6c477fa07e449955, duration=4 ms] 2024-10-16 13:54:11.960 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8e7ea8b2b9efa0f5,5bbca47b7fe39f63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8e7ea8b2b9efa0f5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:11.960 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8e7ea8b2b9efa0f5,5bbca47b7fe39f63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8e7ea8b2b9efa0f5, duration=1 ms] 2024-10-16 13:54:13.018 INFO 4486 --- [ocp-async-9,ae3eef6c30243b1d,8b3ebf93f3cc9c06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:54:13.255 INFO 4486 --- [ocp-schedules-20,5270694c50e6af67,52019c05c8c45998] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:54:13.286 WARN 4486 --- [scanner-13302245-1,5270694c50e6af67,9d48342454624d92] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:54:13.331 INFO 4486 --- [ocp-async-10,aff0676a5e5fe1cf,51c9de70a0c9abb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:13.331 INFO 4486 --- [alarm-detect-2,5270694c50e6af67,47ad4a1130a2aede] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058025 ~ 1729058043 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058040, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058040, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058040=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:54:13.348 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:13.550 INFO 4486 --- [ocp-schedules-21,35ce6f2106d77595,b5b68dc2ae422ae0] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:54:13.552 INFO 4486 --- [ocp-schedules-21,35ce6f2106d77595,b5b68dc2ae422ae0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:13.553 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:54:13.553 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:54:13.553 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:54:13.994 INFO 4486 --- [prometheus-minute-schedule-1,a5baac41930f50c4,2bbc9cc7bf58fdb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:14.002 INFO 4486 --- [prometheus-minute-schedule-1,a5baac41930f50c4,2bbc9cc7bf58fdb5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:14.240 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057980, nodeCount=1000, affectRows=1000 2024-10-16 13:54:14.243 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057980, nodeCount=998, validCount=998 2024-10-16 13:54:14.244 ERROR 4486 --- [metric-archive-second-18,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '586679390261809152-1729057980' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:54:14.243944] [YB420AC531B8-00060B7DAD2B7BD3-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '586679390261809152-1729057980' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:54:14.243944] [YB420AC531B8-00060B7DAD2B7BD3-0-0] 2024-10-16 13:54:14.259 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057980, nodeCount=1000, affectRows=1000 2024-10-16 13:54:14.262 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057980, nodeCount=1000, validCount=1000 2024-10-16 13:54:14.275 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057980, nodeCount=1000, affectRows=1000 2024-10-16 13:54:14.278 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057980, nodeCount=1000, validCount=1000 2024-10-16 13:54:14.461 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729057980, nodeCount=449, affectRows=449 2024-10-16 13:54:14.463 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729057980, nodeCount=448, validCount=448 2024-10-16 13:54:15.099 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729057980, nodeCount=477, affectRows=477 2024-10-16 13:54:15.332 INFO 4486 --- [ocp-async-1,934036a73f1fc84b,1de3e32b8c5b4826] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:15.721 INFO 4486 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729057980, nodeCount=477, validCount=477 2024-10-16 13:54:16.349 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:16.994 INFO 4486 --- [prometheus-minute-schedule-1,73eb88ea91020886,6fbf7245fb681598] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:17.002 INFO 4486 --- [prometheus-minute-schedule-1,73eb88ea91020886,6fbf7245fb681598] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:17.069 INFO 4486 --- [ocp-schedules-26,0a61ea813d152945,3236d3304ebd438c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:54:17.154 INFO 4486 --- [ocp-schedules-13,34b5bb6bb973399f,17be046555bb1bbe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:54:17.222 INFO 4486 --- [ocp-schedules-15,8fb8ceadb5c20548,3ef8acb9ca72108f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:54:17.332 INFO 4486 --- [ocp-async-9,4c9edcc6fa09f720,c7baacfb57319edf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:18.063 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ca732a1a8fbd27e9,bc233f4943dd0b17] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ca732a1a8fbd27e9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:18.063 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ca732a1a8fbd27e9,bc233f4943dd0b17] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ca732a1a8fbd27e9, duration=2 ms] 2024-10-16 13:54:19.160 INFO 4486 --- [ocp-schedules-8,7de43808fdc463fb,edb4cc814f46c0b2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:54:19.332 INFO 4486 --- [ocp-async-3,6245919435ce4e1c,7c53c3ad88a0af1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:19.349 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:19.994 INFO 4486 --- [prometheus-minute-schedule-1,4ae4126a7a489bd8,aac4e6d7489d02ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:20.000 INFO 4486 --- [prometheus-minute-schedule-1,4ae4126a7a489bd8,aac4e6d7489d02ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:20.148 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cda68ef47a1f0ec2,ace18aa204051fa3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cda68ef47a1f0ec2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:20.148 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cda68ef47a1f0ec2,ace18aa204051fa3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cda68ef47a1f0ec2, duration=1 ms] 2024-10-16 13:54:20.813 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:54:20.996 INFO 4486 --- [prometheus-cache-monitor1,482e5ffa60f461b4,bda1e1e905432ac4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:54:20.996 INFO 4486 --- [prometheus-cache-monitor1,482e5ffa60f461b4,bda1e1e905432ac4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:54:21.031 INFO 4486 --- [ocp-schedules-28,042d2ac40c044de2,8086e84336a7dfba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:54:21.332 INFO 4486 --- [ocp-async-10,269de67289f954f2,b0211bfcabdd0fb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:21.689 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4e9a980d8ad62cef,0d5f65a7d9c8f65f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4e9a980d8ad62cef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:21.689 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4e9a980d8ad62cef,0d5f65a7d9c8f65f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4e9a980d8ad62cef, duration=2 ms] 2024-10-16 13:54:21.969 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b3ebe3335e246169,366fb2d199c1593d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b3ebe3335e246169, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:21.969 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b3ebe3335e246169,366fb2d199c1593d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b3ebe3335e246169, duration=1 ms] 2024-10-16 13:54:22.351 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:22.994 INFO 4486 --- [prometheus-minute-schedule-1,4183e704249c6dcb,a5920079fa70ee1b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:23.000 INFO 4486 --- [prometheus-minute-schedule-1,4183e704249c6dcb,a5920079fa70ee1b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:23.018 INFO 4486 --- [ocp-async-8,79c6117bf99c98e7,8ebe61bc2fa75279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:54:23.333 INFO 4486 --- [ocp-async-7,5386ea176eb8d41f,3872ff7e3e9d7bde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:23.468 INFO 4486 --- [ocp-schedules-12,a054dc604f27273b,8ef986c4da5bce14] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:54:23.499 WARN 4486 --- [scanner-13302557-1,a054dc604f27273b,edd843a5e9c023cc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:54:23.556 INFO 4486 --- [ocp-schedules-21,35ce6f2106d77595,b5b68dc2ae422ae0] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:54:23.687 INFO 4486 --- [ocp-schedules-2,520167a9b6ba13c2,07c2e8e656fe466a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:54:24.983 INFO 4486 --- [ocp-schedules-25,398f74bd11a5fe25,602a110903c12db1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:54:25.333 INFO 4486 --- [ocp-async-3,b1967e71e5dac8f3,569616b49f0db3a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:25.352 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:25.994 INFO 4486 --- [prometheus-minute-schedule-1,458526e0dfdf01a3,e2288b8aa9b23b8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:25.999 INFO 4486 --- [prometheus-minute-schedule-1,458526e0dfdf01a3,e2288b8aa9b23b8b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:26.240 INFO 4486 --- [ocp-schedules-30,696deff641a295d4,da092d899f50d81e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:54:26.240 INFO 4486 --- [ocp-schedules-30,696deff641a295d4,da092d899f50d81e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:54:26.272 INFO 4486 --- [Collect all tenants major compactions1,696deff641a295d4,0f662667384035b0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:54:26.272 INFO 4486 --- [Collect all tenants major compactions2,696deff641a295d4,992d8340864e3d07] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:54:26.273 INFO 4486 --- [Collect all tenants major compactions5,696deff641a295d4,7291611688eda988] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:54:26.273 INFO 4486 --- [Collect all tenants major compactions4,696deff641a295d4,fc115c33043bca72] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:54:26.273 INFO 4486 --- [Collect all tenants major compactions3,696deff641a295d4,faf428fc92bffab6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:54:26.286 INFO 4486 --- [Collect all tenants major compactions2,696deff641a295d4,992d8340864e3d07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:26.286 INFO 4486 --- [Collect all tenants major compactions1,696deff641a295d4,0f662667384035b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:26.287 INFO 4486 --- [Collect all tenants major compactions2,696deff641a295d4,992d8340864e3d07] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:54:26.287 INFO 4486 --- [Collect all tenants major compactions1,696deff641a295d4,0f662667384035b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:54:26.290 INFO 4486 --- [Collect all tenants major compactions2,696deff641a295d4,992d8340864e3d07] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:54:26.290 INFO 4486 --- [Collect all tenants major compactions1,696deff641a295d4,0f662667384035b0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:54:26.291 INFO 4486 --- [Collect all tenants major compactions1,696deff641a295d4,0f662667384035b0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:54:26.291 INFO 4486 --- [Collect all tenants major compactions2,696deff641a295d4,992d8340864e3d07] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:54:26.291 INFO 4486 --- [Collect all tenants major compactions3,696deff641a295d4,faf428fc92bffab6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:26.292 INFO 4486 --- [Collect all tenants major compactions3,696deff641a295d4,faf428fc92bffab6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:54:26.296 INFO 4486 --- [Collect all tenants major compactions3,696deff641a295d4,faf428fc92bffab6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:54:26.296 INFO 4486 --- [Collect all tenants major compactions3,696deff641a295d4,faf428fc92bffab6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:54:26.301 INFO 4486 --- [Collect all tenants major compactions5,696deff641a295d4,7291611688eda988] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:26.302 INFO 4486 --- [Collect all tenants major compactions5,696deff641a295d4,7291611688eda988] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:54:26.303 INFO 4486 --- [Collect all tenants major compactions4,696deff641a295d4,fc115c33043bca72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:26.306 INFO 4486 --- [Collect all tenants major compactions4,696deff641a295d4,fc115c33043bca72] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:54:26.307 INFO 4486 --- [Collect all tenants major compactions5,696deff641a295d4,7291611688eda988] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:54:26.307 INFO 4486 --- [Collect all tenants major compactions5,696deff641a295d4,7291611688eda988] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:54:26.308 INFO 4486 --- [Collect all tenants major compactions4,696deff641a295d4,fc115c33043bca72] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:54:26.308 INFO 4486 --- [Collect all tenants major compactions4,696deff641a295d4,fc115c33043bca72] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:54:26.310 INFO 4486 --- [ocp-schedules-30,696deff641a295d4,da092d899f50d81e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:54:27.333 INFO 4486 --- [ocp-async-10,b4f3f1de1e242313,90dfa0374ef9dc1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:27.624 INFO 4486 --- [ocp-schedules-13,f016d01078d62097,cfca5324d0ee7afc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:54:27.624+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:54:27.626 INFO 4486 --- [ocp-schedules-13,f016d01078d62097,cfca5324d0ee7afc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:54:27.626 INFO 4486 --- [ocp-schedules-13,f016d01078d62097,cfca5324d0ee7afc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:54:27.627 INFO 4486 --- [ocp-schedules-13,f016d01078d62097,cfca5324d0ee7afc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:54:27.721 INFO 4486 --- [ocp-schedules-30,6293844bb5d31ec5,9ecd7a99dc5f5664] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:54:27.723 INFO 4486 --- [ocp-schedules-30,6293844bb5d31ec5,9ecd7a99dc5f5664] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:27.760 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:54:27.760 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.764 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:54:27.773 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.774 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:54:27.778 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:54:27.779 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.783 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:54:27.783 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.784 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:54:27.792 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.792 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:54:27.800 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.800 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.800 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:54:27.800 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:54:27.808 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.808 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:27.812 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.817 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.817 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:27.818 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:54:27.819 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.820 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:54:27.823 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:54:27.828 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.828 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:54:27.833 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.834 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:27.835 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.836 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.836 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:54:27.836 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:54:27.846 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.847 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:27.848 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.849 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:54:27.858 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.859 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:54:27.860 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.861 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.861 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:54:27.862 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.863 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.864 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.864 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.865 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.865 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.866 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:54:27.866 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.867 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.868 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.868 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.869 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.869 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:54:27.869 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.870 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.871 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.872 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.872 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.873 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.873 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.874 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.875 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.876 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.876 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.877 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.878 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.878 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.879 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.879 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:54:27.879 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:54:27.886 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:54:27.890 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.892 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:54:27.893 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.894 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.895 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:27.895 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.899 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.900 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.902 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.903 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:27.904 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.904 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:54:27.906 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.906 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.907 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:27.909 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.911 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.914 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:54:27.915 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.918 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.919 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:54:27.919 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:54:27.919 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.919 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.920 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:27.924 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.926 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.930 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.931 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.933 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.935 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.938 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:54:27.941 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:54:27.953 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.954 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.957 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.958 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.961 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.962 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.964 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.965 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.968 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:27.968 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.969 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.970 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.971 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:54:27.972 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.973 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.975 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.976 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:54:27.979 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.980 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:54:27.983 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:27.984 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:54:27.984 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:27.984 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:54:27.985 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:27.985 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:54:27.985 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:27.985 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:54:27.987 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:54:27.988 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:54:27.988 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:54:27.989 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:54:28.039 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:28.040 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:28.041 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:54:28.049 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:28.049 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:54:28.049 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:28.049 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:54:28.049 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:54:28.049 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:54:28.053 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:54:28.053 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:54:28.071 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,58c241cfd4814291,1ed3b8c278e4f67a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=58c241cfd4814291, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:28.071 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,58c241cfd4814291,1ed3b8c278e4f67a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=58c241cfd4814291, duration=2 ms] 2024-10-16 13:54:28.353 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:28.994 INFO 4486 --- [prometheus-minute-schedule-1,048a85f8afc9118a,9c621bdc1aa84fa9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:29.000 INFO 4486 --- [prometheus-minute-schedule-1,048a85f8afc9118a,9c621bdc1aa84fa9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:29.333 INFO 4486 --- [ocp-async-6,0c58bce9491dae3f,c46167f15377f173] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:29.575 INFO 4486 --- [ocp-schedules-16,4ae2d8b025a48603,1cf7a1d1cd478110] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:54:29.575 INFO 4486 --- [ocp-schedules-16,61d0fb0d307e6dfa,0960d2b25261b918] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:54:29.577 INFO 4486 --- [ocp-schedules-16,61d0fb0d307e6dfa,0960d2b25261b918] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:29.607 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.608 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:54:29.619 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.620 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:54:29.642 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.643 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:54:29.645 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:29.645 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.646 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:54:29.685 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:29.685 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.686 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:54:29.687 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:54:29.687 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.690 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:54:29.692 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:54:29.702 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.702 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:54:29.712 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.723 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:29.724 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.725 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:54:29.726 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:54:29.730 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:54:29.735 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:29.736 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:29.750 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:29.752 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.753 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:54:29.761 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:54:29.786 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:54:29.786 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:54:29.804 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:29.805 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.806 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:54:29.809 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.813 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:54:29.816 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:54:29.841 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:54:29.844 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.844 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:54:29.851 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:29.852 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:29.853 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:54:29.864 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:54:29.864 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:54:29.872 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.872 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:54:29.898 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.898 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:54:29.926 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.926 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:54:29.951 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.951 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:54:29.975 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:54:29.998 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:29.999 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:30.000 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:54:30.004 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:54:30.008 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:54:30.012 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:54:30.014 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:54:30.036 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:54:30.103 INFO 4486 --- [prometheus-second-schedule-1,c7aad742cbdd9faf,2e0bd0a718096b9f] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:54:30.158 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c0ca16f67468be12,b689592074c22a60] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c0ca16f67468be12, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:30.158 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c0ca16f67468be12,b689592074c22a60] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c0ca16f67468be12, duration=2 ms] 2024-10-16 13:54:30.186 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:54:30.215 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:30.217 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:30.218 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:54:30.249 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:54:30.249 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:54:31.037 INFO 4486 --- [ocp-schedules-6,aa309f88f8eff49b,8de942009d09bf74] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:54:31.334 INFO 4486 --- [ocp-async-2,120e339878c61701,132552c32a7e1c58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:31.354 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:31.693 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ec133e2e891e586c,301fc0a1aa382974] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ec133e2e891e586c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:31.693 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ec133e2e891e586c,301fc0a1aa382974] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ec133e2e891e586c, duration=0 ms] 2024-10-16 13:54:31.973 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d0c260d325ad1f63,0ee2513751cbc06a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d0c260d325ad1f63, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:31.973 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d0c260d325ad1f63,0ee2513751cbc06a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d0c260d325ad1f63, duration=1 ms] 2024-10-16 13:54:31.994 INFO 4486 --- [prometheus-minute-schedule-1,449d6a5172d0a345,74c9fc18a6031a92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:32.002 INFO 4486 --- [prometheus-minute-schedule-1,449d6a5172d0a345,74c9fc18a6031a92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:33.019 INFO 4486 --- [ocp-async-4,7f2ddc75daeff616,12683a03baf61b32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:54:33.334 INFO 4486 --- [ocp-async-8,8d92967a5fa2df97,5689317324347731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:33.688 INFO 4486 --- [ocp-schedules-18,fed37dd4cb0d540f,ac4698a48732a2d9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:54:33.723 WARN 4486 --- [scanner-13302808-1,fed37dd4cb0d540f,9a6bdf8da2892ca0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:54:33.788 INFO 4486 --- [alarm-detect-2,fed37dd4cb0d540f,f8850348adffdf6d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058040 ~ 1729058063 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058055, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058055, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058055=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:54:34.355 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:34.994 INFO 4486 --- [prometheus-minute-schedule-1,43d9b9de6e4be357,c46a6702cf16c047] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:35.001 INFO 4486 --- [prometheus-minute-schedule-1,43d9b9de6e4be357,c46a6702cf16c047] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:35.334 INFO 4486 --- [ocp-async-6,d90bb97daf29556f,fc2577b503cf9efb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:37.063 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:54:37.065 INFO 4486 --- [ocp-schedules-28,efb062bb15da22d5,5a56e53c26eb4aa3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:54:37.066 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:54:37.066 INFO 4486 --- [ocp-schedules-6,ad2bc333cd1f5e46,3b429465b0e64667] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:54:37.068 INFO 4486 --- [ocp-schedules-6,ad2bc333cd1f5e46,3b429465b0e64667] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:37.069 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:54:37.072 INFO 4486 --- [ocp-schedules-28,efb062bb15da22d5,5a56e53c26eb4aa3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:54:37.081 INFO 4486 --- [ocp-schedules-21,bbb0b95cf29d6300,6837da1b1cb654aa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:54:37.083 INFO 4486 --- [ocp-schedules-21,bbb0b95cf29d6300,6837da1b1cb654aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:54:37.090 INFO 4486 --- [ocp-schedules-4,48db2c579d606e9c,2dc3cb35ae8806df] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:54:37.091 INFO 4486 --- [host-check-agent-status2,48db2c579d606e9c,3b8fee0caa7d7e0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:54:37.091 INFO 4486 --- [host-check-agent-status1,48db2c579d606e9c,8f8db90fc19bc018] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:54:37.091 INFO 4486 --- [host-check-agent-status3,48db2c579d606e9c,802881553234d536] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:54:37.092 INFO 4486 --- [host-check-agent-status4,48db2c579d606e9c,f636c46b37cf75ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:54:37.093 INFO 4486 --- [ocp-schedules-3,28d2a19481dba2f2,dfcfcef468ad3315] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:54:37.093 INFO 4486 --- [ocp-schedules-8,87369168594cef86,03e695fd2cdd1b2f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:54:37.095 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:54:37.095 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:54:37.095 INFO 4486 --- [host-check-clockdiff2,87369168594cef86,59814c102a25b9ef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:54:37.095 INFO 4486 --- [host-check-clockdiff1,87369168594cef86,e381e2faa9ca2bdc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:54:37.097 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:54:37.097 INFO 4486 --- [host-check-clockdiff3,87369168594cef86,20d69665b55e0ce3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:54:37.097 INFO 4486 --- [host-check-clockdiff4,87369168594cef86,71f25693eacf9329] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:54:37.098 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:54:37.100 INFO 4486 --- [host-check-agent-status1,48db2c579d606e9c,8f8db90fc19bc018] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:37.100 INFO 4486 --- [host-check-agent-status2,48db2c579d606e9c,3b8fee0caa7d7e0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:37.103 INFO 4486 --- [host-check-agent-status4,48db2c579d606e9c,f636c46b37cf75ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:37.103 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:37.104 INFO 4486 --- [host-check-agent-status1,48db2c579d606e9c,8f8db90fc19bc018] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:37.104 INFO 4486 --- [host-check-agent-status2,48db2c579d606e9c,3b8fee0caa7d7e0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:37.104 INFO 4486 --- [host-check-agent-status4,48db2c579d606e9c,f636c46b37cf75ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:37.105 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:37.105 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:37.121 INFO 4486 --- [host-check-agent-status3,48db2c579d606e9c,802881553234d536] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:37.121 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:37.121 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:37.132 INFO 4486 --- [host-check-agent-status3,48db2c579d606e9c,802881553234d536] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:37.135 INFO 4486 --- [host-check-clockdiff1,87369168594cef86,e381e2faa9ca2bdc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058077 0 0 , error= 2024-10-16 13:54:37.139 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:37.140 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:37.143 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:37.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:54:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:54:37.164 INFO 4486 --- [host-check-agent-status1,48db2c579d606e9c,8f8db90fc19bc018] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:54:37.164 INFO 4486 --- [host-check-agent-status4,48db2c579d606e9c,f636c46b37cf75ee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:54:37.165 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:54:37.165 INFO 4486 --- [host-check-agent-status3,48db2c579d606e9c,802881553234d536] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:54:37.167 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.168 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:54:37.172244916+08:00 2024-10-16 13:54:37.169 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:54:37.168+08:00 2024-10-16 13:54:37.169 INFO 4486 --- [host-check-agent-status2,48db2c579d606e9c,3b8fee0caa7d7e0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:54:37.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:54:37.169 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:37.170 INFO 4486 --- [host-check-agent-status3,48db2c579d606e9c,802881553234d536] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:54:37.170 INFO 4486 --- [host-check-agent-status3,48db2c579d606e9c,802881553234d536] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:54:37.170 INFO 4486 --- [host-check-agent-status3,48db2c579d606e9c,802881553234d536] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:54:37.170 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:54:37.171 INFO 4486 --- [host-check-agent-status4,48db2c579d606e9c,f636c46b37cf75ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:54:37.171 INFO 4486 --- [host-check-agent-status4,48db2c579d606e9c,f636c46b37cf75ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:54:37.171 INFO 4486 --- [host-check-agent-status4,48db2c579d606e9c,f636c46b37cf75ee] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:54:37.171 INFO 4486 --- [host-check-agent-status1,48db2c579d606e9c,8f8db90fc19bc018] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:54:37.171 INFO 4486 --- [host-check-agent-status1,48db2c579d606e9c,8f8db90fc19bc018] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:54:37.172 INFO 4486 --- [host-check-agent-status1,48db2c579d606e9c,8f8db90fc19bc018] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:54:37.172 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.172 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:54:37.168+08:00 and status AVAILABLE 2024-10-16 13:54:37.172 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:54:37.172202296+08:00 2024-10-16 13:54:37.172 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:54:37.173 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:54:37.172+08:00 2024-10-16 13:54:37.173 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:54:37.173 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:54:37.174 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.174 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:54:37.174 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.175 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:54:37.176 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:54:37.176 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:54:37.172+08:00 and status AVAILABLE 2024-10-16 13:54:37.176 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:37.177 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:54:37.181182452+08:00 2024-10-16 13:54:37.177 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:54:37.181036513+08:00 2024-10-16 13:54:37.177 INFO 4486 --- [host-check-agent-status2,48db2c579d606e9c,3b8fee0caa7d7e0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:54:37.177 INFO 4486 --- [host-check-agent-status2,48db2c579d606e9c,3b8fee0caa7d7e0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:54:37.177 INFO 4486 --- [host-check-agent-status2,48db2c579d606e9c,3b8fee0caa7d7e0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:54:37.177 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:54:37.177+08:00 2024-10-16 13:54:37.178 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:54:37.177+08:00 2024-10-16 13:54:37.178 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.178 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:54:37.180 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:54:37.177+08:00 and status AVAILABLE 2024-10-16 13:54:37.180 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:54:37.177+08:00 and status AVAILABLE 2024-10-16 13:54:37.182 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:54:37.182 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:54:37.182 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:54:37.184 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:37.186 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:37.187 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:37.188 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:37.189 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.189 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.189 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.189 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.189 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.189 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.191 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.191 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:37.192 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:37.192 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:37.193 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:37.193 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:37.204 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.205 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:37.210 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:54:37.211 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:37.212 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:37.217 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.218 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:54:37.219 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:37.220 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.221 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:54:37.223 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:54:37.231 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.232 INFO 4486 --- [ocp-schedules-27,92e83a393b1ce042,18d433a5645ee148] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:54:37.250 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:54:37.254 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:54:37.254 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:54:37.265 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:54:37.265 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:54:37.266 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:54:37.267 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:54:37.276 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:54:37.285 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:37.285 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:37.287 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:37.292 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:54:37.321 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:54:37.323 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:54:37.329 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:54:37.333 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:54:37.334 INFO 4486 --- [ocp-async-5,43d834af3e67fec1,9b31bc8b3f390336] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:37.334 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:54:37.341 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:54:37.346 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:54:37.353 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:37.354 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:37.356 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:37.357 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:54:37.364 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:37.375 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:54:37.383 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:54:37.385 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:54:37.396 INFO 4486 --- [host-check-host-status3,28d2a19481dba2f2,8a442861fd397e1c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:54:37.396 INFO 4486 --- [host-check-host-status4,28d2a19481dba2f2,7e66cf80e0eaebe7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:54:37.405 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:54:37.417 INFO 4486 --- [host-check-host-status2,28d2a19481dba2f2,bd74009d91af07d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:54:37.436 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:54:37.447 INFO 4486 --- [host-check-host-status1,28d2a19481dba2f2,ddab3422dfaf2c27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:54:37.449 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:37.451 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.452 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:54:37.453 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.454 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.455 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:54:37.467 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.467 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.467 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.467 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.467 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.467 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.495 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:54:37.531 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:54:37.581 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:37.596 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:37.600 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.602 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:54:37.603 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:54:37.604 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.605 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:54:37.606 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.607 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.608 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:54:37.617 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.617 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.617 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.617 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.617 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.618 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.626 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:54:37.834 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:54:37.886 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:37.889 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.890 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:54:37.891 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.892 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.893 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:54:37.904 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.904 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.904 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.904 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.904 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:37.904 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:37.921 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:54:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:54:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:54:37.994 INFO 4486 --- [prometheus-minute-schedule-1,13343b9aa6c7e192,d8d57c32c4716d2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:38.003 INFO 4486 --- [prometheus-minute-schedule-1,13343b9aa6c7e192,d8d57c32c4716d2b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:38.081 INFO 4486 --- [host-check-clockdiff3,87369168594cef86,20d69665b55e0ce3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058078 4 4 , error= 2024-10-16 13:54:38.081 INFO 4486 --- [host-check-clockdiff2,87369168594cef86,59814c102a25b9ef] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058078 5 5 , error= 2024-10-16 13:54:38.081 INFO 4486 --- [host-check-clockdiff4,87369168594cef86,71f25693eacf9329] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058078 4 4 , error= 2024-10-16 13:54:38.093 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a5525e22c42a9765,1b1a86f34daa6033] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a5525e22c42a9765, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:38.093 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a5525e22c42a9765,1b1a86f34daa6033] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a5525e22c42a9765, duration=1 ms] 2024-10-16 13:54:38.172 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:54:38.216 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:54:38.220 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:38.221 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:54:38.222 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:38.222 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:38.223 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:54:38.233 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:38.233 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:38.233 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:38.233 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:38.233 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:54:38.234 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:38.295 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:54:38.551 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:54:38.301656, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:54:38.551 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:54:38.301656, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:54:38.588 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:54:38.301656} } 2024-10-16 13:54:38.638 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:54:38.648 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:54:38.652 INFO 4486 --- [ocp-schedules-11,b4a7fa9cbc17a1a4,2fbc03feaf23050d] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:54:39.165 INFO 4486 --- [ocp-schedules-8,26d92642cd19e316,0b032ecae5a498c6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:54:39.334 INFO 4486 --- [ocp-async-10,46ab844a7a5fbe19,9d56b81b1ff60a40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:40.006 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:54:40.006 INFO 4486 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:54:40.025 INFO 4486 --- [Check obproxy1,787a0fd0add747fd,d9dd5e26f937ca13] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:54:40.026 INFO 4486 --- [Check obproxy1,787a0fd0add747fd,d9dd5e26f937ca13] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:40.028 INFO 4486 --- [Check obproxy2,f2c9fa6d96fe1084,8c65908a5fe88f26] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:54:40.028 INFO 4486 --- [Check obproxy2,f2c9fa6d96fe1084,8c65908a5fe88f26] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:40.029 INFO 4486 --- [Collect obproxy2,a20b4796b584bd6e,5d9ab3d600309f81] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:54:40.029 INFO 4486 --- [Collect obproxy2,a20b4796b584bd6e,5d9ab3d600309f81] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:40.031 INFO 4486 --- [Collect obproxy1,91041a4d65bc2e62,064338fc36a9969b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:54:40.031 INFO 4486 --- [Collect obproxy1,91041a4d65bc2e62,064338fc36a9969b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:54:40.051 INFO 4486 --- [Check obproxy2,f2c9fa6d96fe1084,8c65908a5fe88f26] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:54:40.051 INFO 4486 --- [Check obproxy2,f2c9fa6d96fe1084,8c65908a5fe88f26] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:54:40.052 INFO 4486 --- [Collect obproxy2,a20b4796b584bd6e,5d9ab3d600309f81] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:54:40.053 INFO 4486 --- [Check obproxy1,787a0fd0add747fd,d9dd5e26f937ca13] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:54:40.053 INFO 4486 --- [Check obproxy1,787a0fd0add747fd,d9dd5e26f937ca13] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:54:40.056 INFO 4486 --- [Collect obproxy1,91041a4d65bc2e62,064338fc36a9969b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:54:40.058 INFO 4486 --- [Collect obproxy2,a20b4796b584bd6e,5d9ab3d600309f81] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:54:40.062 INFO 4486 --- [Collect obproxy1,91041a4d65bc2e62,064338fc36a9969b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:54:40.064 INFO 4486 --- [Collect obproxy2,a20b4796b584bd6e,5d9ab3d600309f81] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:54:40.065 INFO 4486 --- [Collect obproxy1,91041a4d65bc2e62,064338fc36a9969b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:54:40.163 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3fb08192e71929e7,2bc118f7a3b9086a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3fb08192e71929e7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:40.163 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3fb08192e71929e7,2bc118f7a3b9086a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3fb08192e71929e7, duration=2 ms] 2024-10-16 13:54:40.357 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:40.994 INFO 4486 --- [prometheus-minute-schedule-1,ee57d51ca79469dc,ece13a453978cf5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:40.995 INFO 4486 --- [prometheus-hour-schedule-1,91ac9e0f8646d1ac,63a9bfc36484690c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:54:41.001 INFO 4486 --- [prometheus-minute-schedule-1,ee57d51ca79469dc,ece13a453978cf5b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:41.043 INFO 4486 --- [ocp-schedules-9,64ac9074ee4f8ec1,33c00b8799036228] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:54:41.334 INFO 4486 --- [ocp-async-6,f0731e8e5f8a7284,687b7f95b85f39d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:41.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c46383bbadf538c0,ca589dc99f71282b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c46383bbadf538c0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:41.701 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,c46383bbadf538c0,ca589dc99f71282b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c46383bbadf538c0, duration=1 ms] 2024-10-16 13:54:41.978 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1cc908e127e1f796,8ab48176268ec354] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1cc908e127e1f796, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:41.979 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1cc908e127e1f796,8ab48176268ec354] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1cc908e127e1f796, duration=1 ms] 2024-10-16 13:54:42.159 INFO 4486 --- [ocp-schedules-28,b25f77745b421062,3bdedfa66c25942f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:54:43.019 INFO 4486 --- [ocp-async-9,0359c7ed53485508,3daee2a995eb4b11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:54:43.088 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 13:54:43.088 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 13:54:43.090 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.090 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:54:43.092 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.093 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.093 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.094 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 13:54:43.111 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.112 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.113 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.113 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 13:54:43.128 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.129 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.130 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.130 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 13:54:43.145 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.146 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.146 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.147 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 13:54:43.161 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.162 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.163 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.163 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 13:54:43.170 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.171 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.172 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.172 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 13:54:43.178 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.178 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.179 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.180 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 13:54:43.186 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.186 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.187 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.188 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 13:54:43.195 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.195 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.196 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.196 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 13:54:43.202 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.203 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.203 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.204 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 13:54:43.209 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.210 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.210 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.211 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 13:54:43.220 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.220 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.221 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.221 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 13:54:43.236 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.236 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.237 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.237 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 13:54:43.242 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.243 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.244 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.244 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 13:54:43.250 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.250 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.251 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.251 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 13:54:43.258 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.259 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.259 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.260 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 13:54:43.265 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.266 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.267 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.267 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 13:54:43.271 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.272 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.272 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.273 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 13:54:43.277 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.278 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.279 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.279 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 13:54:43.285 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.286 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.287 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.287 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 13:54:43.294 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.294 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.295 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.296 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 13:54:43.302 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.303 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.304 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.304 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 13:54:43.310 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.311 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.311 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.312 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 13:54:43.320 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.321 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.322 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.322 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 13:54:43.328 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.328 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.329 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.329 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 13:54:43.334 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.335 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.335 INFO 4486 --- [ocp-async-4,dc9416e0073b2e2a,afd3d09ca2999146] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:43.335 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.336 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 13:54:43.342 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.342 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.343 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.344 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 13:54:43.349 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.350 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.351 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.351 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 13:54:43.356 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.357 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.358 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:43.358 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.359 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 13:54:43.364 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.365 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.366 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.366 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 13:54:43.371 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.372 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.373 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.373 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 13:54:43.378 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.379 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.379 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.380 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 13:54:43.385 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.385 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.386 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.387 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 13:54:43.391 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.392 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.392 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.393 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 13:54:43.401 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.401 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.402 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.402 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 13:54:43.410 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.410 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.411 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.411 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 13:54:43.417 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.417 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.418 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.419 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 13:54:43.424 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.424 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.425 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.426 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 13:54:43.430 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.431 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.431 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.432 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 13:54:43.436 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.436 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.437 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.437 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 13:54:43.442 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.443 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.444 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.444 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 13:54:43.449 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.449 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.451 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.451 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 13:54:43.461 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.462 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.462 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.463 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 13:54:43.484 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.485 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.486 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.486 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 13:54:43.493 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.494 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.495 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.496 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 13:54:43.503 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.504 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.505 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.506 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 13:54:43.514 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.514 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.515 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.516 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 13:54:43.520 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.520 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.521 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.521 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 13:54:43.526 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.526 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.527 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.527 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 13:54:43.532 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 13:54:43.532 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 13:54:43.533 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.534 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:54:43.535 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.536 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.537 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.537 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 13:54:43.545 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.545 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 13:54:43.546 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:54:43.547 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 13:54:43.552 INFO 4486 --- [ocp-schedules-23,4c1eaa0c1e0afacf,4862159d4f1dcc86] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 13:54:43.860 INFO 4486 --- [ocp-schedules-7,d5dbdfae7fc59de8,078e8b8f2069dc47] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:54:43.892 WARN 4486 --- [scanner-13303124-1,d5dbdfae7fc59de8,0573a98ede494393] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:54:43.963 INFO 4486 --- [alarm-detect-15,d5dbdfae7fc59de8,cbd47295daf69805] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058055 ~ 1729058073 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058070, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058070, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058070=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:54:43.995 INFO 4486 --- [prometheus-minute-schedule-1,a523510d03e46728,c5a09f90e482e50c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:44.010 INFO 4486 --- [prometheus-minute-schedule-1,a523510d03e46728,c5a09f90e482e50c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:45.335 INFO 4486 --- [ocp-async-1,20121075d2b64a8d,9b07e684b56c3ef2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:45.558 INFO 4486 --- [ocp-schedules-5,d265fedf470c5c8d,27024d061826fd60] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:54:45.563 INFO 4486 --- [ocp-schedules-5,d265fedf470c5c8d,27024d061826fd60] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:54:45.563 INFO 4486 --- [ocp-schedules-5,d265fedf470c5c8d,27024d061826fd60] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:54:45.589 INFO 4486 --- [ocp-schedules-4,253a310268b5c735,91d28646ffdafc07] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:54:45.589 INFO 4486 --- [ocp-schedules-4,253a310268b5c735,91d28646ffdafc07] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:54:46.359 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:46.994 INFO 4486 --- [prometheus-minute-schedule-1,3dc610d9135d6214,b626733468816e92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:47.000 INFO 4486 --- [prometheus-minute-schedule-1,3dc610d9135d6214,b626733468816e92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:47.070 INFO 4486 --- [ocp-schedules-6,ad2bc333cd1f5e46,3b429465b0e64667] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:54:47.174 INFO 4486 --- [ocp-schedules-21,bbb0b95cf29d6300,6837da1b1cb654aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:54:47.225 INFO 4486 --- [ocp-schedules-29,57376ec72d39b288,f027a783f165c8b8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:54:47.335 INFO 4486 --- [ocp-async-7,2390e3ee253211e1,5394ebe75d184816] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:48.099 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1b81a0e941780274,fd079c394240c5e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1b81a0e941780274, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:48.099 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1b81a0e941780274,fd079c394240c5e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1b81a0e941780274, duration=1 ms] 2024-10-16 13:54:49.335 INFO 4486 --- [ocp-async-5,e5c7abaad64b66cb,d4e10cb3990fc07a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:49.359 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:49.994 INFO 4486 --- [prometheus-minute-schedule-1,1ffbfb443bb30bc8,dfaf282ed7ecce84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:50.002 INFO 4486 --- [prometheus-minute-schedule-1,1ffbfb443bb30bc8,dfaf282ed7ecce84] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:50.168 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c25725a26702bd2b,29ed600d2e127bbd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c25725a26702bd2b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:50.168 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c25725a26702bd2b,29ed600d2e127bbd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c25725a26702bd2b, duration=1 ms] 2024-10-16 13:54:50.519 INFO 4486 --- [ocp-schedules-28,715ffcb9a94888e3,7f00f405eba6e5da] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:54:50.519 INFO 4486 --- [ocp-schedules-28,715ffcb9a94888e3,7f00f405eba6e5da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:54:50.520 INFO 4486 --- [ocp-schedules-28,715ffcb9a94888e3,7f00f405eba6e5da] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:54:50.520 INFO 4486 --- [ocp-schedules-28,715ffcb9a94888e3,7f00f405eba6e5da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:54:50.521 INFO 4486 --- [ocp-schedules-28,715ffcb9a94888e3,7f00f405eba6e5da] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:54:50.521 INFO 4486 --- [ocp-schedules-28,715ffcb9a94888e3,7f00f405eba6e5da] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:54:50.791 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:54:50.791 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:54:50.804 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:54:50.804 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:54:50.804 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:54:50.817 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:54:50.817 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:54:50.817 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:54:50.824 INFO 4486 --- [ocp-schedules-7,498ccebfa26ea5f3,c1bde999a7c9e81c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:54:50.868 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fb3bff0f029eea65,95ce3e7bab16e3c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=fb3bff0f029eea65, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:54:50.869 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fb3bff0f029eea65,95ce3e7bab16e3c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=fb3bff0f029eea65, duration=1 ms] 2024-10-16 13:54:51.049 INFO 4486 --- [ocp-schedules-23,83d0a00bd31be253,c5c4b455acaef941] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:54:51.123 INFO 4486 --- [ocp-schedules-17,51c3bb60bf9c42f0,7b2d44c392ef5e76] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:54:51.123 INFO 4486 --- [ocp-schedules-17,51c3bb60bf9c42f0,7b2d44c392ef5e76] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:54:51.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:54:51.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:54:51.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:54:51.124 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:54:51.123 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:54:51.128 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:54:51.129 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:54:51.129 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:54:51.129 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:54:51.129 INFO 4486 --- [ocp-schedules-17,51c3bb60bf9c42f0,7b2d44c392ef5e76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:54:51.130 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:54:51.130 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:54:51.130 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:54:51.130 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:54:51.131 INFO 4486 --- [ocp-schedules-17,51c3bb60bf9c42f0,7b2d44c392ef5e76] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:54:51.131 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:54:51.132 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:54:51.152 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:54:51.153 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:54:51.155 INFO 4486 --- [ocp-schedules-17,51c3bb60bf9c42f0,7b2d44c392ef5e76] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:54:51.155 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:54:51.156 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:54:51.160 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:54:51.162 INFO 4486 --- [ocp-schedules-17,51c3bb60bf9c42f0,7b2d44c392ef5e76] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:54:51.336 INFO 4486 --- [ocp-async-1,7104a910a25169fa,7f9b9753c5409956] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:51.709 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d7ce25773d68ed7a,a7f351c7f2e7c46c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d7ce25773d68ed7a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:51.709 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d7ce25773d68ed7a,a7f351c7f2e7c46c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d7ce25773d68ed7a, duration=1 ms] 2024-10-16 13:54:51.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,bd9aec3a843bede8,8d96ee7a9ba771ab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bd9aec3a843bede8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:51.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,bd9aec3a843bede8,8d96ee7a9ba771ab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bd9aec3a843bede8, duration=1 ms] 2024-10-16 13:54:52.360 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:52.994 INFO 4486 --- [prometheus-minute-schedule-1,8574d583f7227789,be3f7592b2e9bc69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:53.002 INFO 4486 --- [prometheus-minute-schedule-1,8574d583f7227789,be3f7592b2e9bc69] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:53.019 INFO 4486 --- [ocp-async-8,dd9e5858eccb027c,d8b8fb86c7aad827] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:54:53.336 INFO 4486 --- [ocp-async-6,761514fe6e482fd4,59bc46edc4e94650] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:54.013 INFO 4486 --- [ocp-schedules-12,5c5ea12d4ba82c68,50814e2c92bca0ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:54:54.062 WARN 4486 --- [scanner-13303348-1,5c5ea12d4ba82c68,a9b8fefd84201444] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:54:54.917 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,18ca9184d2a79f07,b43e0f46e7c2ccad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=18ca9184d2a79f07, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:54:54.918 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,18ca9184d2a79f07,b43e0f46e7c2ccad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=18ca9184d2a79f07, duration=3 ms] 2024-10-16 13:54:55.336 INFO 4486 --- [ocp-async-5,e694a91bcdf98259,013c0788eb9cc863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:55.361 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:55.994 INFO 4486 --- [prometheus-minute-schedule-1,5cd66af735226a18,fd85ac367dec3324] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:56.001 INFO 4486 --- [prometheus-minute-schedule-1,5cd66af735226a18,fd85ac367dec3324] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:57.336 INFO 4486 --- [ocp-async-1,2bd4516540bd80f0,c75ca1e42da54d3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:54:58.103 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d0a2b60fc94e2f88,6b14b9da6ff8fe3b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d0a2b60fc94e2f88, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:54:58.103 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d0a2b60fc94e2f88,6b14b9da6ff8fe3b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d0a2b60fc94e2f88, duration=1 ms] 2024-10-16 13:54:58.362 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:54:58.910 INFO 4486 --- [ocp-schedules-1,897c3a6b917a7dfa,8de495274b698a85] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:54:58.912 INFO 4486 --- [ocp-schedules-1,897c3a6b917a7dfa,8de495274b698a85] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:54:58.994 INFO 4486 --- [prometheus-minute-schedule-1,79d067453b47873c,0dd35df7156c4e69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058100, interval=60 2024-10-16 13:54:58.999 INFO 4486 --- [prometheus-minute-schedule-1,79d067453b47873c,0dd35df7156c4e69] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:54:59.169 INFO 4486 --- [ocp-schedules-11,99b104e1cc221b01,8fb42bef45da80c4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:54:59.336 INFO 4486 --- [ocp-async-9,aa284ae1e4ba1699,f3486f14b29603b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:00.172 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0beeb5993a97360f,17ceaf8b6daff007] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0beeb5993a97360f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:00.172 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0beeb5993a97360f,17ceaf8b6daff007] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0beeb5993a97360f, duration=1 ms] 2024-10-16 13:55:00.246 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:50:00.0, nextRun=2024-10-16 13:55:00.0, currentDate=Wed Oct 16 13:55:00 CST 2024 2024-10-16 13:55:00.246 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:50:00.0, nextRunTime=2024-10-16 13:55:00.0} 2024-10-16 13:55:00.250 INFO 4486 --- [ocp-schedules-16,61d0fb0d307e6dfa,0960d2b25261b918] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:55:00.250 INFO 4486 --- [ocp-schedules-16,f7c7a3c60f018e84,80c7588586113758] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:55:00.250 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-10-16 13:55:00.250 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.197.49.184', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-10-16T13:55:00.250+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='791e68b70ade5cad', operation=EXECUTE} 2024-10-16 13:55:00.252 INFO 4486 --- [ocp-schedules-16,f7c7a3c60f018e84,80c7588586113758] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:55:00.266 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:50:00.0, nextRun=2024-10-16 13:55:00.0, currentDate=Wed Oct 16 13:55:00 CST 2024 2024-10-16 13:55:00.266 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 13:50:00.0, nextRunTime=2024-10-16 13:55:00.0} 2024-10-16 13:55:00.286 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.287 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:00.292 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.294 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:00.295 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 1000003] 2024-10-16 13:55:00.310 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.p.s.internal.SqlParseTaskSchedule : The last sql parse service is still running, skip current task 2024-10-16 13:55:00.310 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,3a74e140b4696f2e] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-10-16 13:55:00.322 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.323 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:00.331 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:00.332 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.332 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:55:00.370 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:00.371 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.371 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:00.373 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:55:00.373 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.376 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:55:00.377 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:00.385 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.386 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:00.397 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.406 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:00.407 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.407 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:00.408 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:55:00.413 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:55:00.419 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:00.420 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:00.449 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:55:00.455 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:55:00.456 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:55:00.467 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:00.468 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,57c3249ad37a4c8f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7933729, operation:EXECUTE, state: RUNNING 2024-10-16 13:55:00.469 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.470 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:55:00.482 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7933729, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266495, task_operation=execute, latest_execution_start_time=2024-10-16T13:55:00.461+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=7933729}, listMap={}}, executor=10.197.49.184 2024-10-16 13:55:00.517 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.532 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:00.534 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.535 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:55:00.535 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:00.538 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.542 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:55:00.547 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:00.548 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:00.548 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.549 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:55:00.560 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:55:00.560 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:55:00.578 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.578 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:00.585 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.607 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.607 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:00.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:00.677 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.677 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:00.707 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.708 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:00.724 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.752 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:00.771 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.782 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for OceanBase 2024-10-16 13:55:00.782 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.ocp.diagnosis.model.NeuralNetwork : W:[1079.4750314957316], b:[-1.850753682719265E12] 2024-10-16 13:55:00.787 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.788 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:00.790 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:00.791 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:00.795 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:55:00.801 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:55:00.806 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:55:00.808 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:55:00.840 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:55:00.856 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.887 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.914 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 13:55:00.975 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:55:00.979 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.006 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:01.009 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:01.010 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:55:01.018 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.048 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:55:01.048 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:55:01.055 INFO 4486 --- [ocp-schedules-22,4cdf16bc4777ad44,87178ee4b5de0a24] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:55:01.303 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.337 INFO 4486 --- [ocp-async-2,f7e9ba0846c39266,5e06f828c6629aa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:01.341 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.363 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:01.384 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for obce_3zones 2024-10-16 13:55:01.384 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1295.9606763372929], b:[2.2870925145727866E12] 2024-10-16 13:55:01.389 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.468 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ad7af1c360c5227c,603738b656187267] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=ad7af1c360c5227c, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 13:55:01.537 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.570 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.600 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 13:55:01.697 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ad7af1c360c5227c,603738b656187267] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=ad7af1c360c5227c, duration=240 ms] 2024-10-16 13:55:01.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e348d0b09e000112,a9765a2b0b6dfb42] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e348d0b09e000112, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:01.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e348d0b09e000112,a9765a2b0b6dfb42] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e348d0b09e000112, duration=1 ms] 2024-10-16 13:55:01.741 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:01.779 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:01.974 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:01.988 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f2bb7a28b5f8ae9d,909623ea7d5cf02f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f2bb7a28b5f8ae9d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:01.988 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f2bb7a28b5f8ae9d,909623ea7d5cf02f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f2bb7a28b5f8ae9d, duration=1 ms] 2024-10-16 13:55:01.994 INFO 4486 --- [prometheus-minute-schedule-1,11a11ec3d998fe50,8c5a9c87b7641a7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:01.999 INFO 4486 --- [prometheus-minute-schedule-1,11a11ec3d998fe50,8c5a9c87b7641a7c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:02.006 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:02.028 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for glob_3zone 2024-10-16 13:55:02.028 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1292.3504806553328], b:[2.281317553551766E12] 2024-10-16 13:55:02.033 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:02.130 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:02.164 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:02.192 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 13:55:02.296 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,394b2b7b16b20ed7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7933729, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 13:55:02.321 INFO 4486 --- [ocp-schedules-4,e1e9f22355247241,bf40d0627ef5ed84] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:55:02.321+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:55:02.324 INFO 4486 --- [ocp-schedules-4,e1e9f22355247241,bf40d0627ef5ed84] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:55:02.329 INFO 4486 --- [ocp-schedules-4,e1e9f22355247241,bf40d0627ef5ed84] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:55:02.341 INFO 4486 --- [ocp-schedules-4,e1e9f22355247241,bf40d0627ef5ed84] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:55:03.019 INFO 4486 --- [ocp-async-3,c54fbd893fdf6350,217e22c7765ad59b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:55:03.337 INFO 4486 --- [ocp-async-1,78a325e79cd7868c,64df88bf39f684b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:04.232 INFO 4486 --- [ocp-schedules-7,2a951471cd7c63c8,5bb1edb058841687] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:55:04.268 WARN 4486 --- [scanner-13304000-1,2a951471cd7c63c8,4be3e84903f500d2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:55:04.342 INFO 4486 --- [alarm-detect-7,2a951471cd7c63c8,08ad2b860d40cd9b] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058070 ~ 1729058094 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058085, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058085, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058085=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:55:04.364 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:04.994 INFO 4486 --- [prometheus-minute-schedule-1,7d628d68f3f92ae7,beb5bef60a205c6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:05.000 INFO 4486 --- [prometheus-minute-schedule-1,7d628d68f3f92ae7,beb5bef60a205c6f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:05.337 INFO 4486 --- [ocp-async-7,24b58fff13776e82,0c60c384919b1f66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:05.804 INFO 4486 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:5266495, status:SUCCESS 2024-10-16 13:55:05.805 INFO 4486 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:5266495, status:SUCCESS 2024-10-16 13:55:07.066 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:55:07.066 INFO 4486 --- [ocp-schedules-7,9da9aadc4b2ffdbe,264cb0bf166e0119] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:55:07.069 INFO 4486 --- [ocp-schedules-7,9da9aadc4b2ffdbe,264cb0bf166e0119] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:55:07.081 INFO 4486 --- [ocp-schedules-6,c1436406fc2d60f0,14dbf53ee4f6bee4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:55:07.083 INFO 4486 --- [ocp-schedules-6,c1436406fc2d60f0,14dbf53ee4f6bee4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:55:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:55:07.090 INFO 4486 --- [ocp-schedules-13,3cdcac474c3d277b,9e4ca243ed16d2a3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:55:07.091 INFO 4486 --- [ocp-schedules-25,6e78c84fc12ef6d0,db9a3a47924495f1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:55:07.091 INFO 4486 --- [host-check-agent-status2,3cdcac474c3d277b,47563712de3b287a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:55:07.091 INFO 4486 --- [host-check-agent-status1,3cdcac474c3d277b,6b9af654e9aaedf1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:55:07.092 INFO 4486 --- [host-check-agent-status4,3cdcac474c3d277b,b9374d5cb52d06b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:55:07.092 INFO 4486 --- [host-check-agent-status3,3cdcac474c3d277b,6c4fbc4e94c66c79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:55:07.092 INFO 4486 --- [ocp-schedules-19,21cc8e41dfeca8ba,5852875c09c2ecad] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:55:07.092 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:55:07.092 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:55:07.092 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:55:07.093 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:55:07.093 INFO 4486 --- [host-check-clockdiff1,21cc8e41dfeca8ba,08f05963cdb41c83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:55:07.093 INFO 4486 --- [host-check-clockdiff2,21cc8e41dfeca8ba,8f74f71191a9697c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:55:07.094 INFO 4486 --- [host-check-clockdiff3,21cc8e41dfeca8ba,2a26b745be3f5e04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:55:07.095 INFO 4486 --- [host-check-clockdiff4,21cc8e41dfeca8ba,7e947ff7ffc7973c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:55:07.102 INFO 4486 --- [host-check-agent-status3,3cdcac474c3d277b,6c4fbc4e94c66c79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:55:07.102 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:55:07.102 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:55:07.102 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:55:07.102 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:07.102 INFO 4486 --- [host-check-agent-status1,3cdcac474c3d277b,6b9af654e9aaedf1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:07.103 INFO 4486 --- [host-check-agent-status2,3cdcac474c3d277b,47563712de3b287a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:55:07.103 INFO 4486 --- [host-check-agent-status4,3cdcac474c3d277b,b9374d5cb52d06b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:55:07.112 INFO 4486 --- [host-check-agent-status2,3cdcac474c3d277b,47563712de3b287a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:55:07.112 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:07.112 INFO 4486 --- [host-check-agent-status1,3cdcac474c3d277b,6b9af654e9aaedf1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:07.112 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:55:07.112 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:55:07.112 INFO 4486 --- [host-check-agent-status3,3cdcac474c3d277b,6c4fbc4e94c66c79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:55:07.112 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:55:07.116 INFO 4486 --- [host-check-agent-status4,3cdcac474c3d277b,b9374d5cb52d06b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:55:07.119 INFO 4486 --- [host-check-clockdiff1,21cc8e41dfeca8ba,08f05963cdb41c83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058107 0 0 , error= 2024-10-16 13:55:07.124 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.125 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:07.126 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:55:07.150 INFO 4486 --- [host-check-agent-status1,3cdcac474c3d277b,6b9af654e9aaedf1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:55:07.150 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.151 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:55:07.151 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:55:07.151 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:07.151 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:55:07.152 INFO 4486 --- [host-check-agent-status4,3cdcac474c3d277b,b9374d5cb52d06b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:55:07.152 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:55:07.153 INFO 4486 --- [host-check-agent-status3,3cdcac474c3d277b,6c4fbc4e94c66c79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:55:07.154 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:55:07.158393478+08:00 2024-10-16 13:55:07.154 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:55:07.158135499+08:00 2024-10-16 13:55:07.155 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:55:07.159172123+08:00 2024-10-16 13:55:07.154 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:55:07.153584976+08:00 2024-10-16 13:55:07.155 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:55:07.154+08:00 2024-10-16 13:55:07.155 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:55:07.154+08:00 2024-10-16 13:55:07.155 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:55:07.155+08:00 2024-10-16 13:55:07.155 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:55:07.155+08:00 2024-10-16 13:55:07.156 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-agent-status1,3cdcac474c3d277b,6b9af654e9aaedf1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-agent-status1,3cdcac474c3d277b,6b9af654e9aaedf1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-agent-status1,3cdcac474c3d277b,6b9af654e9aaedf1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-agent-status3,3cdcac474c3d277b,6c4fbc4e94c66c79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-agent-status3,3cdcac474c3d277b,6c4fbc4e94c66c79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-agent-status3,3cdcac474c3d277b,6c4fbc4e94c66c79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:55:07.155+08:00 and status AVAILABLE 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:55:07.154+08:00 and status AVAILABLE 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:55:07.154+08:00 and status AVAILABLE 2024-10-16 13:55:07.158 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:55:07.155+08:00 and status AVAILABLE 2024-10-16 13:55:07.159 INFO 4486 --- [host-check-agent-status2,3cdcac474c3d277b,47563712de3b287a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:55:07.160 INFO 4486 --- [host-check-agent-status4,3cdcac474c3d277b,b9374d5cb52d06b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:55:07.160 INFO 4486 --- [host-check-agent-status4,3cdcac474c3d277b,b9374d5cb52d06b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:55:07.160 INFO 4486 --- [host-check-agent-status4,3cdcac474c3d277b,b9374d5cb52d06b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:55:07.161 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:55:07.163 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:55:07.163 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:55:07.163 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:55:07.164 INFO 4486 --- [host-check-agent-status2,3cdcac474c3d277b,47563712de3b287a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:55:07.164 INFO 4486 --- [host-check-agent-status2,3cdcac474c3d277b,47563712de3b287a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:55:07.164 INFO 4486 --- [host-check-agent-status2,3cdcac474c3d277b,47563712de3b287a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:55:07.167 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.171 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:55:07.171 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:55:07.171 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:07.172 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:55:07.172 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:55:07.172 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:55:07.173 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:55:07.173 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:07.173 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:55:07.181 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.182 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:55:07.191 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.192 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:55:07.197 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:07.197 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:07.197 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:07.197 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:07.204 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.205 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:55:07.207 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.208 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:55:07.219 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:07.220 INFO 4486 --- [ocp-schedules-17,6735daca61cca134,fe48829509acaa90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:55:07.228 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:55:07.232 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:55:07.235 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:55:07.241 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:55:07.244 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:55:07.248 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:55:07.250 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:55:07.259 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:07.261 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:07.262 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:55:07.265 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:07.278 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:07.287 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:55:07.297 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:55:07.298 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:55:07.303 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:55:07.307 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:55:07.314 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:55:07.315 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:07.322 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:07.331 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:07.337 INFO 4486 --- [ocp-async-2,02bc3b045cdb955f,f4d9ab0467f3504b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:07.343 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:55:07.345 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:55:07.353 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:55:07.354 INFO 4486 --- [host-check-host-status3,6e78c84fc12ef6d0,76c1ad3fd14a875e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:55:07.355 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:55:07.363 INFO 4486 --- [host-check-host-status4,6e78c84fc12ef6d0,a75cfb536cfb8b8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:55:07.365 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:07.368 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:55:07.374 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:07.378 INFO 4486 --- [host-check-host-status2,6e78c84fc12ef6d0,abf2903af1a116dd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:55:07.424 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:55:07.434 INFO 4486 --- [host-check-host-status1,6e78c84fc12ef6d0,1a7ce576b84eba16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:55:07.740 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,184dbf305f65217e,12d8d9e414d4c954] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=184dbf305f65217e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:55:07.741 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,184dbf305f65217e,12d8d9e414d4c954] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=184dbf305f65217e, duration=3 ms] 2024-10-16 13:55:07.994 INFO 4486 --- [prometheus-minute-schedule-1,f932e5f0770415b6,faa16c62af7d0fd3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:08.000 INFO 4486 --- [prometheus-minute-schedule-1,f932e5f0770415b6,faa16c62af7d0fd3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:08.073 INFO 4486 --- [host-check-clockdiff3,21cc8e41dfeca8ba,2a26b745be3f5e04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058108 5 5 , error= 2024-10-16 13:55:08.080 INFO 4486 --- [host-check-clockdiff4,21cc8e41dfeca8ba,7e947ff7ffc7973c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058108 4 4 , error= 2024-10-16 13:55:08.081 INFO 4486 --- [host-check-clockdiff2,21cc8e41dfeca8ba,8f74f71191a9697c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058108 4 4 , error= 2024-10-16 13:55:08.109 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2a61b7c10c29218e,93267451646ebcb5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2a61b7c10c29218e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:08.109 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2a61b7c10c29218e,93267451646ebcb5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2a61b7c10c29218e, duration=1 ms] 2024-10-16 13:55:08.940 INFO 4486 --- [ocp-schedules-1,897c3a6b917a7dfa,8de495274b698a85] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:55:09.337 INFO 4486 --- [ocp-async-4,56b58b84b0c4bd44,288a04cfee81a358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:10.006 INFO 4486 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:55:10.007 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:55:10.025 INFO 4486 --- [Check obproxy2,5cfcb76aaae4e078,8314471a753dea3e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:55:10.026 INFO 4486 --- [Check obproxy1,764d2e15d6eb3673,624ff1ca120fbeec] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:55:10.026 INFO 4486 --- [Check obproxy2,5cfcb76aaae4e078,8314471a753dea3e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:10.026 INFO 4486 --- [Check obproxy1,764d2e15d6eb3673,624ff1ca120fbeec] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:10.030 INFO 4486 --- [Collect obproxy1,e0c68d0bdf2fc1a7,3444a5633eb89e31] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:55:10.030 INFO 4486 --- [Collect obproxy1,e0c68d0bdf2fc1a7,3444a5633eb89e31] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:10.033 INFO 4486 --- [Collect obproxy2,9d3bde28d1174140,37cd2cd4972f21a8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:55:10.033 INFO 4486 --- [Collect obproxy2,9d3bde28d1174140,37cd2cd4972f21a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:10.053 INFO 4486 --- [Collect obproxy2,9d3bde28d1174140,37cd2cd4972f21a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:55:10.055 INFO 4486 --- [Check obproxy2,5cfcb76aaae4e078,8314471a753dea3e] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:55:10.055 INFO 4486 --- [Check obproxy2,5cfcb76aaae4e078,8314471a753dea3e] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:55:10.057 INFO 4486 --- [Collect obproxy2,9d3bde28d1174140,37cd2cd4972f21a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:55:10.061 INFO 4486 --- [Check obproxy1,764d2e15d6eb3673,624ff1ca120fbeec] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:55:10.061 INFO 4486 --- [Check obproxy1,764d2e15d6eb3673,624ff1ca120fbeec] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:55:10.063 INFO 4486 --- [Collect obproxy2,9d3bde28d1174140,37cd2cd4972f21a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:55:10.067 INFO 4486 --- [Collect obproxy1,e0c68d0bdf2fc1a7,3444a5633eb89e31] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:55:10.073 INFO 4486 --- [Collect obproxy1,e0c68d0bdf2fc1a7,3444a5633eb89e31] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:55:10.078 INFO 4486 --- [Collect obproxy1,e0c68d0bdf2fc1a7,3444a5633eb89e31] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:55:10.176 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,76b2319329c59626,52f79d25b696785b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=76b2319329c59626, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:10.176 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,76b2319329c59626,52f79d25b696785b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=76b2319329c59626, duration=1 ms] 2024-10-16 13:55:10.366 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:55:10.994 INFO 4486 --- [prometheus-minute-schedule-1,abff4257aeed8db9,91926663b3880c31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:10.999 INFO 4486 --- [prometheus-minute-schedule-1,abff4257aeed8db9,91926663b3880c31] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:55:11.061 INFO 4486 --- [ocp-schedules-8,7a8b9d5cb49e229d,ff6407c8a6bd9427] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:55:11.338 INFO 4486 --- [ocp-async-9,3696434e21d87e22,6d5f2fbad3341268] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:11.718 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7d5b0319ab82a71a,10a9b0fa7ac0e421] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7d5b0319ab82a71a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:11.718 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7d5b0319ab82a71a,10a9b0fa7ac0e421] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7d5b0319ab82a71a, duration=1 ms] 2024-10-16 13:55:11.995 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4e3a0154254b3c7b,7e4d47a3bb1c6086] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4e3a0154254b3c7b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:11.995 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4e3a0154254b3c7b,7e4d47a3bb1c6086] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4e3a0154254b3c7b, duration=1 ms] 2024-10-16 13:55:13.019 INFO 4486 --- [ocp-async-1,5027b8a3687e0cba,fb97b2bc7f037164] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:55:13.338 INFO 4486 --- [ocp-async-6,39984d72ed9222e3,d0eebe6a0958458e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:13.367 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:13.994 INFO 4486 --- [prometheus-minute-schedule-1,22aa319cb932f054,e01531c7d829da2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:14.005 INFO 4486 --- [prometheus-minute-schedule-1,22aa319cb932f054,e01531c7d829da2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:14.279 ERROR 4486 --- [metric-archive-second-10,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '669741699544190976-1729058040' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:55:14.277665] [YB420AC531B8-00060B7D84EAB2EC-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '669741699544190976-1729058040' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:55:14.277665] [YB420AC531B8-00060B7D84EAB2EC-0-0] 2024-10-16 13:55:14.306 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058040, nodeCount=1000, affectRows=1000 2024-10-16 13:55:14.308 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058040, nodeCount=1000, affectRows=1000 2024-10-16 13:55:14.311 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058040, nodeCount=998, validCount=998 2024-10-16 13:55:14.311 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058040, nodeCount=1000, validCount=1000 2024-10-16 13:55:14.330 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058040, nodeCount=1000, affectRows=1000 2024-10-16 13:55:14.333 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058040, nodeCount=1000, validCount=1000 2024-10-16 13:55:14.430 INFO 4486 --- [ocp-schedules-9,f6e58cefffa09041,91d81d0abdad0f9b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:55:14.458 WARN 4486 --- [scanner-13305042-1,f6e58cefffa09041,02ffe3d9b6502ab4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:55:14.520 INFO 4486 --- [alarm-detect-3,f6e58cefffa09041,b7d428df71feba9b] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058085 ~ 1729058104 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058100, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058100, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058100=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:55:14.560 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058040, nodeCount=450, affectRows=450 2024-10-16 13:55:14.562 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058040, nodeCount=449, validCount=449 2024-10-16 13:55:15.144 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058040, nodeCount=480, affectRows=480 2024-10-16 13:55:15.219 INFO 4486 --- [ocp-schedules-11,ea6d98579368c1b9,2a13e5a515aa3060] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 13:55:15.219 INFO 4486 --- [ocp-schedules-11,ea6d98579368c1b9,2a13e5a515aa3060] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 13:55:15.250 INFO 4486 --- [pool-282-thread-1,ea6d98579368c1b9,6abd7f81d789b456] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:55:15.250 INFO 4486 --- [pool-282-thread-2,ea6d98579368c1b9,8bd6538e19a46ad0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:55:15.251 INFO 4486 --- [pool-282-thread-3,ea6d98579368c1b9,b4c526e8c97b0ba4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 13:55:15.273 INFO 4486 --- [pool-282-thread-1,ea6d98579368c1b9,6abd7f81d789b456] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:15.275 INFO 4486 --- [pool-282-thread-2,ea6d98579368c1b9,8bd6538e19a46ad0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:15.279 INFO 4486 --- [pool-282-thread-1,ea6d98579368c1b9,6abd7f81d789b456] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058115000000] 2024-10-16 13:55:15.279 INFO 4486 --- [pool-282-thread-2,ea6d98579368c1b9,8bd6538e19a46ad0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058115000000] 2024-10-16 13:55:15.281 INFO 4486 --- [pool-282-thread-3,ea6d98579368c1b9,b4c526e8c97b0ba4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:15.285 INFO 4486 --- [pool-282-thread-3,ea6d98579368c1b9,b4c526e8c97b0ba4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058115000000] 2024-10-16 13:55:15.338 INFO 4486 --- [ocp-async-5,0cd49194497c5372,8977caba8aa0a351] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:15.560 WARN 4486 --- [pool-282-thread-3,ea6d98579368c1b9,b4c526e8c97b0ba4] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 13:55:15.560 WARN 4486 --- [pool-282-thread-3,ea6d98579368c1b9,b4c526e8c97b0ba4] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 13:55:15.562 INFO 4486 --- [pool-282-thread-3,ea6d98579368c1b9,b4c526e8c97b0ba4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:55:15.795 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058040, nodeCount=480, validCount=480 2024-10-16 13:55:15.905 WARN 4486 --- [pool-282-thread-1,ea6d98579368c1b9,6abd7f81d789b456] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 13:55:15.905 WARN 4486 --- [pool-282-thread-1,ea6d98579368c1b9,6abd7f81d789b456] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1009 2024-10-16 13:55:15.907 INFO 4486 --- [pool-282-thread-1,ea6d98579368c1b9,6abd7f81d789b456] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:55:16.063 WARN 4486 --- [pool-282-thread-2,ea6d98579368c1b9,8bd6538e19a46ad0] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 13:55:16.064 INFO 4486 --- [pool-282-thread-2,ea6d98579368c1b9,8bd6538e19a46ad0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 13:55:16.065 INFO 4486 --- [ocp-schedules-11,ea6d98579368c1b9,2a13e5a515aa3060] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 13:55:16.368 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:16.994 INFO 4486 --- [prometheus-minute-schedule-1,e74343f3a4ae3707,e3913d63caa53cfc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:17.000 INFO 4486 --- [prometheus-minute-schedule-1,e74343f3a4ae3707,e3913d63caa53cfc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:17.070 INFO 4486 --- [ocp-schedules-7,9da9aadc4b2ffdbe,264cb0bf166e0119] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:55:17.156 INFO 4486 --- [ocp-schedules-6,c1436406fc2d60f0,14dbf53ee4f6bee4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:55:17.228 INFO 4486 --- [ocp-schedules-17,4126ce0a808017f7,89d98ffd3f3da6e6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:55:17.338 INFO 4486 --- [ocp-async-10,8fc50a25677a5cdf,0d9e506935a6aaed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:18.113 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4588d3ef97098c77,02dbaa1192637f52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4588d3ef97098c77, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:18.113 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,4588d3ef97098c77,02dbaa1192637f52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4588d3ef97098c77, duration=1 ms] 2024-10-16 13:55:19.174 INFO 4486 --- [ocp-schedules-22,647a74ef9a5cd267,cbd70bc4c12225b3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:55:19.338 INFO 4486 --- [ocp-async-3,c9e6c7b6f5de977c,d3fa2210ac8ddd73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:19.368 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:19.994 INFO 4486 --- [prometheus-minute-schedule-1,64a059ecc5c4a33b,734c6dfcfd8554ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:19.999 INFO 4486 --- [prometheus-minute-schedule-1,64a059ecc5c4a33b,734c6dfcfd8554ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:20.180 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b131d9bdbd2a4125,465569e7155c2f74] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b131d9bdbd2a4125, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:20.180 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b131d9bdbd2a4125,465569e7155c2f74] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b131d9bdbd2a4125, duration=1 ms] 2024-10-16 13:55:20.996 INFO 4486 --- [prometheus-cache-monitor1,2588751ffeb33c13,99432a4eec8558da] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:55:20.996 INFO 4486 --- [prometheus-cache-monitor1,2588751ffeb33c13,99432a4eec8558da] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:55:21.067 INFO 4486 --- [ocp-schedules-7,d291cb35908976a1,b4cceafdc0954436] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:55:21.339 INFO 4486 --- [ocp-async-8,b7d9ce678748343c,6e17b721908c63b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:21.724 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,158d959968401e07,07a387ba6d0b4c3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=158d959968401e07, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:21.724 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,158d959968401e07,07a387ba6d0b4c3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=158d959968401e07, duration=1 ms] 2024-10-16 13:55:22.002 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b3ba6324d37d372e,3f4b7fe0a3582cdc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b3ba6324d37d372e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:22.002 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b3ba6324d37d372e,3f4b7fe0a3582cdc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b3ba6324d37d372e, duration=1 ms] 2024-10-16 13:55:22.369 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:22.994 INFO 4486 --- [prometheus-minute-schedule-1,094a253ec3999a88,14da35b8857172df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:22.999 INFO 4486 --- [prometheus-minute-schedule-1,094a253ec3999a88,14da35b8857172df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:23.020 INFO 4486 --- [ocp-async-2,f382019db9050c30,43e72bf1a1987ef0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:55:23.339 INFO 4486 --- [ocp-async-10,c7b23764bfb5a05c,a64b5cc11587acd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:23.557 INFO 4486 --- [ocp-schedules-9,cbb6638821a4963f,60ab07f9f3b94e24] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:55:23.560 INFO 4486 --- [ocp-schedules-9,cbb6638821a4963f,60ab07f9f3b94e24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:55:23.561 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:55:23.561 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:55:23.561 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:55:23.720 INFO 4486 --- [ocp-schedules-5,50e0c893e0d1268a,893817ef7d3ae5b9] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:55:24.599 INFO 4486 --- [ocp-schedules-29,2cafbae82813209d,713ef8844db33b20] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:55:24.627 WARN 4486 --- [scanner-13305384-1,2cafbae82813209d,e6043eb01286de18] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:55:25.017 INFO 4486 --- [ocp-schedules-19,6d706bf2f0955210,97e9d5947e6c198d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:55:25.339 INFO 4486 --- [ocp-async-3,192aecacfdf9a983,4a96434925e012cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:25.370 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:25.994 INFO 4486 --- [prometheus-minute-schedule-1,d696b25f6dffb551,6168687a039ec0ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:26.003 INFO 4486 --- [prometheus-minute-schedule-1,d696b25f6dffb551,6168687a039ec0ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:26.310 INFO 4486 --- [ocp-schedules-10,c65fca94c6c6b720,c526ceb4c907f29c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:55:26.310 INFO 4486 --- [ocp-schedules-10,c65fca94c6c6b720,c526ceb4c907f29c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:55:26.344 INFO 4486 --- [Collect all tenants major compactions4,c65fca94c6c6b720,cf7ea3727297e5cf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:55:26.344 INFO 4486 --- [Collect all tenants major compactions1,c65fca94c6c6b720,d49013c2eeddc737] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:55:26.344 INFO 4486 --- [Collect all tenants major compactions2,c65fca94c6c6b720,f8d7dfe38741a076] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:55:26.345 INFO 4486 --- [Collect all tenants major compactions3,c65fca94c6c6b720,a9e81079f9a13850] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:55:26.345 INFO 4486 --- [Collect all tenants major compactions5,c65fca94c6c6b720,8fc86cf99dfe0fff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:55:26.357 INFO 4486 --- [Collect all tenants major compactions2,c65fca94c6c6b720,f8d7dfe38741a076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:26.357 INFO 4486 --- [Collect all tenants major compactions1,c65fca94c6c6b720,d49013c2eeddc737] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:26.358 INFO 4486 --- [Collect all tenants major compactions1,c65fca94c6c6b720,d49013c2eeddc737] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:55:26.358 INFO 4486 --- [Collect all tenants major compactions2,c65fca94c6c6b720,f8d7dfe38741a076] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:55:26.362 INFO 4486 --- [Collect all tenants major compactions2,c65fca94c6c6b720,f8d7dfe38741a076] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:55:26.362 INFO 4486 --- [Collect all tenants major compactions1,c65fca94c6c6b720,d49013c2eeddc737] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:55:26.362 INFO 4486 --- [Collect all tenants major compactions2,c65fca94c6c6b720,f8d7dfe38741a076] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:55:26.362 INFO 4486 --- [Collect all tenants major compactions1,c65fca94c6c6b720,d49013c2eeddc737] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:55:26.365 INFO 4486 --- [Collect all tenants major compactions3,c65fca94c6c6b720,a9e81079f9a13850] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:26.368 INFO 4486 --- [Collect all tenants major compactions3,c65fca94c6c6b720,a9e81079f9a13850] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:55:26.372 INFO 4486 --- [Collect all tenants major compactions3,c65fca94c6c6b720,a9e81079f9a13850] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:55:26.372 INFO 4486 --- [Collect all tenants major compactions3,c65fca94c6c6b720,a9e81079f9a13850] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:55:26.379 INFO 4486 --- [Collect all tenants major compactions5,c65fca94c6c6b720,8fc86cf99dfe0fff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:26.379 INFO 4486 --- [Collect all tenants major compactions4,c65fca94c6c6b720,cf7ea3727297e5cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:26.381 INFO 4486 --- [Collect all tenants major compactions5,c65fca94c6c6b720,8fc86cf99dfe0fff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:55:26.381 INFO 4486 --- [Collect all tenants major compactions4,c65fca94c6c6b720,cf7ea3727297e5cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:55:26.385 INFO 4486 --- [Collect all tenants major compactions4,c65fca94c6c6b720,cf7ea3727297e5cf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:55:26.385 INFO 4486 --- [Collect all tenants major compactions4,c65fca94c6c6b720,cf7ea3727297e5cf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:55:26.386 INFO 4486 --- [Collect all tenants major compactions5,c65fca94c6c6b720,8fc86cf99dfe0fff] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:55:26.386 INFO 4486 --- [Collect all tenants major compactions5,c65fca94c6c6b720,8fc86cf99dfe0fff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:55:26.387 INFO 4486 --- [ocp-schedules-10,c65fca94c6c6b720,c526ceb4c907f29c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:55:27.339 INFO 4486 --- [ocp-async-8,6125400e800bcaf8,d2777c329da4a906] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:27.657 INFO 4486 --- [ocp-schedules-6,383a60824e2862eb,850fff5221f5e426] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:55:27.657+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:55:27.659 INFO 4486 --- [ocp-schedules-6,383a60824e2862eb,850fff5221f5e426] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:55:27.660 INFO 4486 --- [ocp-schedules-6,383a60824e2862eb,850fff5221f5e426] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:55:27.661 INFO 4486 --- [ocp-schedules-6,383a60824e2862eb,850fff5221f5e426] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:55:28.053 INFO 4486 --- [ocp-schedules-30,6293844bb5d31ec5,9ecd7a99dc5f5664] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:55:28.119 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d26289824d5c5cb4,9bbc78cf31a3a5c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d26289824d5c5cb4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:28.119 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d26289824d5c5cb4,9bbc78cf31a3a5c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d26289824d5c5cb4, duration=2 ms] 2024-10-16 13:55:28.371 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:28.994 INFO 4486 --- [prometheus-minute-schedule-1,c2fe83869bc05c86,0c0d4e8e78f9802b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:29.001 INFO 4486 --- [prometheus-minute-schedule-1,c2fe83869bc05c86,0c0d4e8e78f9802b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:29.339 INFO 4486 --- [ocp-async-4,0079621e06b687a5,5a78bd2edfd99221] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:30.103 INFO 4486 --- [prometheus-second-schedule-1,deca58e40a32b806,416eb43fe9d94a09] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:55:30.188 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c34e002315573dcc,3822dd83ce2480cb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c34e002315573dcc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:30.188 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c34e002315573dcc,3822dd83ce2480cb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c34e002315573dcc, duration=1 ms] 2024-10-16 13:55:31.049 INFO 4486 --- [ocp-schedules-16,f7c7a3c60f018e84,80c7588586113758] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:55:31.049 INFO 4486 --- [ocp-schedules-16,e5e826675c61b41f,8efacf683307f4d4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:55:31.051 INFO 4486 --- [ocp-schedules-16,e5e826675c61b41f,8efacf683307f4d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:55:31.073 INFO 4486 --- [ocp-schedules-18,e6ca77a806569086,f7711eb48468cb5a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:55:31.083 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.084 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:31.095 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.096 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:31.121 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.122 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.122 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:55:31.124 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.125 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:31.152 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.152 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.153 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:31.154 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:55:31.154 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.157 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:55:31.159 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:31.169 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.170 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:31.181 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.191 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.191 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.192 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:31.193 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:55:31.200 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:55:31.206 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:31.209 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:31.232 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.235 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.236 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:55:31.239 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:55:31.257 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:55:31.257 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:55:31.287 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.289 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.289 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:55:31.293 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.297 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:55:31.300 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:31.314 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:55:31.325 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.326 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.326 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:31.326 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.327 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:55:31.336 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:55:31.336 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:55:31.340 INFO 4486 --- [ocp-async-9,24ac5404fa0407f2,80f66e49ad6f95bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:31.354 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.354 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:31.372 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:31.380 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.380 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:31.409 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.409 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:55:31.435 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.435 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:55:31.467 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:55:31.494 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.496 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.497 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:31.501 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:55:31.506 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:55:31.511 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:55:31.512 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:55:31.537 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:55:31.682 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:55:31.708 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:31.710 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:31.711 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:55:31.730 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0df67012db5bddaa,ec85f2b5a0a29073] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0df67012db5bddaa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:31.730 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0df67012db5bddaa,ec85f2b5a0a29073] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0df67012db5bddaa, duration=2 ms] 2024-10-16 13:55:31.737 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:55:31.737 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:55:31.994 INFO 4486 --- [prometheus-minute-schedule-1,3101dc52cce2fd58,3add33894699cdd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:32.001 INFO 4486 --- [prometheus-minute-schedule-1,3101dc52cce2fd58,3add33894699cdd7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:32.007 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,378341bbaceaf9e5,9a8bd47e604bacc6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=378341bbaceaf9e5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:32.007 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,378341bbaceaf9e5,9a8bd47e604bacc6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=378341bbaceaf9e5, duration=1 ms] 2024-10-16 13:55:32.254 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e5ecbf2f2b838e28,6f7a3997e9d92b64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=e5ecbf2f2b838e28, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:55:32.255 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e5ecbf2f2b838e28,6f7a3997e9d92b64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=e5ecbf2f2b838e28, duration=1 ms] 2024-10-16 13:55:33.020 INFO 4486 --- [ocp-async-10,ec15cfcd703b130b,35b729a6dc47798b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:55:33.340 INFO 4486 --- [ocp-async-6,17886cac0b4b67fa,58271e73941de7e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:33.561 INFO 4486 --- [ocp-schedules-9,cbb6638821a4963f,60ab07f9f3b94e24] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:55:34.372 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:34.743 INFO 4486 --- [ocp-schedules-11,9ef823851d83c009,db807d09ddcd615d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:55:34.771 WARN 4486 --- [scanner-13305583-1,9ef823851d83c009,9d450327614dd55f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:55:34.836 INFO 4486 --- [alarm-detect-8,9ef823851d83c009,b6527aee1c76fe64] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058100 ~ 1729058124 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058115, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058115, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058115=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:55:34.994 INFO 4486 --- [prometheus-minute-schedule-1,44ae717e1dab71c4,03a280c548e7c05b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:35.000 INFO 4486 --- [prometheus-minute-schedule-1,44ae717e1dab71c4,03a280c548e7c05b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:35.340 INFO 4486 --- [ocp-async-5,b71d7b945bf3ef9e,8cd95a0fb42b9f3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:37.063 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:55:37.064 INFO 4486 --- [ocp-schedules-28,0bded4a19fa3c3ad,5b0927f97fe18e51] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:55:37.066 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:55:37.066 INFO 4486 --- [ocp-schedules-24,49983aee1b92e676,d3db9329136a2ca2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:55:37.067 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:55:37.068 INFO 4486 --- [ocp-schedules-24,49983aee1b92e676,d3db9329136a2ca2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:55:37.072 INFO 4486 --- [ocp-schedules-28,0bded4a19fa3c3ad,5b0927f97fe18e51] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:55:37.112 INFO 4486 --- [ocp-schedules-30,a0164dee6a46951f,1e6ec8b0c3a8f5fa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:55:37.114 INFO 4486 --- [ocp-schedules-30,a0164dee6a46951f,1e6ec8b0c3a8f5fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:55:37.115 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:55:37.120 INFO 4486 --- [ocp-schedules-11,1f1818ec255ef9b2,b7bbc44a845320f1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:55:37.121 INFO 4486 --- [ocp-schedules-22,3dd99d84359d5e80,65b4a5e2b95128ee] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:55:37.121 INFO 4486 --- [host-check-agent-status2,3dd99d84359d5e80,3ddac8aaa8efa1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:55:37.121 INFO 4486 --- [host-check-agent-status1,3dd99d84359d5e80,21539b070a1d7207] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:55:37.122 INFO 4486 --- [host-check-agent-status3,3dd99d84359d5e80,b7639699fa042dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:55:37.122 INFO 4486 --- [host-check-clockdiff1,1f1818ec255ef9b2,e50e00846b50b5e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:55:37.122 INFO 4486 --- [host-check-clockdiff2,1f1818ec255ef9b2,af3201951dcf860e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:55:37.122 INFO 4486 --- [host-check-clockdiff4,1f1818ec255ef9b2,f3737b34a9e6fc52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:55:37.122 INFO 4486 --- [ocp-schedules-17,16f742165e4d5baf,f388d20be423847e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:55:37.122 INFO 4486 --- [host-check-clockdiff3,1f1818ec255ef9b2,4944651b47501840] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:55:37.123 INFO 4486 --- [host-check-agent-status4,3dd99d84359d5e80,04588976937a36ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:55:37.124 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:55:37.124 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:55:37.124 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:55:37.125 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:55:37.129 INFO 4486 --- [host-check-agent-status4,3dd99d84359d5e80,04588976937a36ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:55:37.130 INFO 4486 --- [host-check-agent-status2,3dd99d84359d5e80,3ddac8aaa8efa1dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:55:37.136 INFO 4486 --- [host-check-agent-status1,3dd99d84359d5e80,21539b070a1d7207] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:37.137 INFO 4486 --- [host-check-agent-status3,3dd99d84359d5e80,b7639699fa042dbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:55:37.138 INFO 4486 --- [host-check-agent-status4,3dd99d84359d5e80,04588976937a36ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:55:37.151 INFO 4486 --- [host-check-agent-status3,3dd99d84359d5e80,b7639699fa042dbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:55:37.151 INFO 4486 --- [host-check-agent-status2,3dd99d84359d5e80,3ddac8aaa8efa1dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:55:37.151 INFO 4486 --- [host-check-agent-status1,3dd99d84359d5e80,21539b070a1d7207] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:37.151 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:55:37.152 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:55:37.152 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:55:37.159 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:37.172 INFO 4486 --- [host-check-clockdiff1,1f1818ec255ef9b2,e50e00846b50b5e5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058137 0 0 , error= 2024-10-16 13:55:37.172 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:55:37.172 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:55:37.172 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:55:37.173 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.174 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:37.174 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:37.175 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:55:37.195 INFO 4486 --- [host-check-agent-status4,3dd99d84359d5e80,04588976937a36ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:55:37.196 INFO 4486 --- [host-check-agent-status1,3dd99d84359d5e80,21539b070a1d7207] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:55:37.196 INFO 4486 --- [host-check-agent-status3,3dd99d84359d5e80,b7639699fa042dbb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:55:37.199 INFO 4486 --- [host-check-agent-status2,3dd99d84359d5e80,3ddac8aaa8efa1dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:55:37.200 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.200 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:55:37.201 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:55:37.201 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status1,3dd99d84359d5e80,21539b070a1d7207] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status3,3dd99d84359d5e80,b7639699fa042dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status3,3dd99d84359d5e80,b7639699fa042dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status3,3dd99d84359d5e80,b7639699fa042dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status1,3dd99d84359d5e80,21539b070a1d7207] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status1,3dd99d84359d5e80,21539b070a1d7207] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status4,3dd99d84359d5e80,04588976937a36ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status4,3dd99d84359d5e80,04588976937a36ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-agent-status4,3dd99d84359d5e80,04588976937a36ed] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:55:37.202 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:55:37.206583429+08:00 2024-10-16 13:55:37.203 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:55:37.202+08:00 2024-10-16 13:55:37.203 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:55:37.208185736+08:00 2024-10-16 13:55:37.204 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:55:37.204 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:55:37.203+08:00 2024-10-16 13:55:37.205 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:55:37.204767188+08:00 2024-10-16 13:55:37.205 INFO 4486 --- [host-check-agent-status2,3dd99d84359d5e80,3ddac8aaa8efa1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:55:37.205 INFO 4486 --- [host-check-agent-status2,3dd99d84359d5e80,3ddac8aaa8efa1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:55:37.205 INFO 4486 --- [host-check-agent-status2,3dd99d84359d5e80,3ddac8aaa8efa1dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:55:37.205 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:55:37.205 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:55:37.205+08:00 2024-10-16 13:55:37.206 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:55:37.206 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:55:37.210564424+08:00 2024-10-16 13:55:37.206 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:55:37.202+08:00 and status AVAILABLE 2024-10-16 13:55:37.206 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:55:37.203+08:00 and status AVAILABLE 2024-10-16 13:55:37.207 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:55:37.206+08:00 2024-10-16 13:55:37.208 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:55:37.205+08:00 and status AVAILABLE 2024-10-16 13:55:37.209 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:55:37.209 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:55:37.209 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:55:37.206+08:00 and status AVAILABLE 2024-10-16 13:55:37.210 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.210 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:55:37.211 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:55:37.211 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.212 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:55:37.213 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.213 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.214 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:55:37.216 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:55:37.217 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:55:37.218 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:55:37.218 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:37.219 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:55:37.219 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:55:37.220 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:37.222 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:55:37.224 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:55:37.225 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.226 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:55:37.227 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.227 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.227 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.227 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.227 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.227 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.235 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.236 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:55:37.242 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:37.243 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:37.245 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.246 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:37.246 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:55:37.246 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:55:37.249 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.250 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:55:37.251 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:55:37.264 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.265 INFO 4486 --- [ocp-schedules-2,a7a2823680b1a1d8,686d30601548dfc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:55:37.270 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:55:37.281 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:55:37.282 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:55:37.283 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:55:37.292 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:55:37.293 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:55:37.300 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:55:37.302 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:37.309 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:37.310 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:37.311 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:55:37.328 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:55:37.335 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:55:37.340 INFO 4486 --- [ocp-async-7,f5731892094cbaaa,ed175a775d509760] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:37.348 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:55:37.352 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:55:37.360 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:55:37.364 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:55:37.367 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:37.372 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:55:37.374 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:37.382 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:37.388 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:37.390 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:55:37.397 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:55:37.401 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:55:37.408 INFO 4486 --- [host-check-host-status3,16f742165e4d5baf,956709498f6d5b3d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:55:37.414 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:55:37.417 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:55:37.426 INFO 4486 --- [host-check-host-status4,16f742165e4d5baf,c3abd76183483de6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:55:37.430 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:55:37.442 INFO 4486 --- [host-check-host-status2,16f742165e4d5baf,2f7ad1feee3193af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:55:37.473 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:55:37.476 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:37.478 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.479 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:55:37.480 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.482 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.482 INFO 4486 --- [host-check-host-status1,16f742165e4d5baf,81acbc5c6a55479c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:55:37.483 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:55:37.497 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.497 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.497 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.497 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.497 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.497 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.525 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:55:37.563 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:55:37.610 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:37.627 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:37.629 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.630 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:55:37.631 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:55:37.632 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.634 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:55:37.635 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.635 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.636 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:55:37.648 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.648 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.648 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.648 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.648 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.649 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.660 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:55:37.842 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:55:37.886 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:37.888 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.889 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:55:37.890 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.891 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.892 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:55:37.901 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.901 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.901 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.901 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.901 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:37.902 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:37.919 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:55:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:55:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:55:37.994 INFO 4486 --- [prometheus-minute-schedule-1,e56706f64a7d5973,55b686303b7ccd49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:38.000 INFO 4486 --- [prometheus-minute-schedule-1,e56706f64a7d5973,55b686303b7ccd49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:38.089 INFO 4486 --- [host-check-clockdiff2,1f1818ec255ef9b2,af3201951dcf860e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058138 5 5 , error= 2024-10-16 13:55:38.093 INFO 4486 --- [host-check-clockdiff3,1f1818ec255ef9b2,4944651b47501840] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058138 4 4 , error= 2024-10-16 13:55:38.124 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,96d11cd4c7863789,3fa6d8caa342cf84] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=96d11cd4c7863789, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:38.124 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,96d11cd4c7863789,3fa6d8caa342cf84] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=96d11cd4c7863789, duration=1 ms] 2024-10-16 13:55:38.141 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:55:38.190 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:55:38.193 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:38.193 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:55:38.195 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:38.195 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:38.196 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:55:38.207 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:38.207 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:38.207 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:38.207 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:38.207 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:55:38.208 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:55:38.270 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:55:38.477 INFO 4486 --- [ocp-schedules-21,3e0d797d4f0fd764,68de19c3f351ed13] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13799,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:55:38.477 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:55:38.275478, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:55:38.477 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:55:38.275478, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:55:38.478 INFO 4486 --- [ocp-schedules-21,3e0d797d4f0fd764,68de19c3f351ed13] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:55:38.523 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:55:38.275478} } 2024-10-16 13:55:38.582 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:55:38.589 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:55:38.592 INFO 4486 --- [ocp-schedules-12,d815f1a2b80cabda,902c6ac17ca4833c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:55:38.682 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,11c84aec53ce0b89,76d0fcf5fbc1adda] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=11c84aec53ce0b89, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:55:38.682 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,11c84aec53ce0b89,76d0fcf5fbc1adda] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=11c84aec53ce0b89, duration=1 ms] 2024-10-16 13:55:39.178 INFO 4486 --- [ocp-schedules-1,96557cb69853ee18,c46544687fdbbf6c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:55:39.341 INFO 4486 --- [ocp-async-8,3d17feadef92e730,d0fc1a3db7d26124] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:40.006 INFO 4486 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:55:40.006 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:55:40.024 INFO 4486 --- [Check obproxy1,c6517a3e405d8c9f,f12bd5c40224e962] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:55:40.025 INFO 4486 --- [Check obproxy1,c6517a3e405d8c9f,f12bd5c40224e962] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:40.025 INFO 4486 --- [Check obproxy2,ab401a330e4c5efa,b175f086e141d92b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:55:40.026 INFO 4486 --- [Check obproxy2,ab401a330e4c5efa,b175f086e141d92b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:40.026 INFO 4486 --- [Collect obproxy2,de8929c97969df96,cedce29b19cc7a82] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:55:40.026 INFO 4486 --- [Collect obproxy2,de8929c97969df96,cedce29b19cc7a82] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:40.028 INFO 4486 --- [Collect obproxy1,5f94fe8495c1b8e1,42f5c091c878e6c4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:55:40.028 INFO 4486 --- [Collect obproxy1,5f94fe8495c1b8e1,42f5c091c878e6c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:55:40.048 INFO 4486 --- [Check obproxy1,c6517a3e405d8c9f,f12bd5c40224e962] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:55:40.048 INFO 4486 --- [Check obproxy1,c6517a3e405d8c9f,f12bd5c40224e962] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:55:40.048 INFO 4486 --- [Check obproxy2,ab401a330e4c5efa,b175f086e141d92b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:55:40.048 INFO 4486 --- [Check obproxy2,ab401a330e4c5efa,b175f086e141d92b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:55:40.048 INFO 4486 --- [Collect obproxy1,5f94fe8495c1b8e1,42f5c091c878e6c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:55:40.049 INFO 4486 --- [Collect obproxy2,de8929c97969df96,cedce29b19cc7a82] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:55:40.052 INFO 4486 --- [Collect obproxy1,5f94fe8495c1b8e1,42f5c091c878e6c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:55:40.054 INFO 4486 --- [Collect obproxy2,de8929c97969df96,cedce29b19cc7a82] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:55:40.055 INFO 4486 --- [Collect obproxy1,5f94fe8495c1b8e1,42f5c091c878e6c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:55:40.059 INFO 4486 --- [Collect obproxy2,de8929c97969df96,cedce29b19cc7a82] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:55:40.195 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ac875061e73e53b5,67562ab04d67b4bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ac875061e73e53b5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:40.195 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ac875061e73e53b5,67562ab04d67b4bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ac875061e73e53b5, duration=1 ms] 2024-10-16 13:55:40.375 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:40.994 INFO 4486 --- [prometheus-minute-schedule-1,c5dc66a0e6fe058c,9692b7e1bcf16973] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:40.995 INFO 4486 --- [prometheus-hour-schedule-1,083d26764a887d01,86f5717f7835bb10] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:55:41.002 INFO 4486 --- [prometheus-minute-schedule-1,c5dc66a0e6fe058c,9692b7e1bcf16973] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:41.080 INFO 4486 --- [ocp-schedules-25,1852d059ecad8423,0351315c521de2a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:55:41.277 INFO 4486 --- [host-check-clockdiff4,1f1818ec255ef9b2,f3737b34a9e6fc52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058141 4 4 , error= 2024-10-16 13:55:41.341 INFO 4486 --- [ocp-async-4,6d9ffbcede15b961,fd0e0c9499f5ee60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:41.735 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,79dcd0d15adef9e5,82f8e8457f1ff9ee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=79dcd0d15adef9e5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:41.735 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,79dcd0d15adef9e5,82f8e8457f1ff9ee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=79dcd0d15adef9e5, duration=1 ms] 2024-10-16 13:55:42.013 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3bbef5e5d057f7c9,cc78b0472c489ea6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3bbef5e5d057f7c9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:42.013 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3bbef5e5d057f7c9,cc78b0472c489ea6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3bbef5e5d057f7c9, duration=1 ms] 2024-10-16 13:55:42.166 INFO 4486 --- [ocp-schedules-6,eb2de76fdcb39e75,26ff70931a248dbf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:55:43.020 INFO 4486 --- [ocp-async-5,b5f3f7d3c6697218,b03de2df0892e057] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:55:43.341 INFO 4486 --- [ocp-async-3,c7bde1393ad8dc1f,30c898ed913e62cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:43.375 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:43.994 INFO 4486 --- [prometheus-minute-schedule-1,7330230e63b6cff4,f3ed008602ba88a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:44.001 INFO 4486 --- [prometheus-minute-schedule-1,7330230e63b6cff4,f3ed008602ba88a6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:44.926 INFO 4486 --- [ocp-schedules-15,f8f1dcc5a5599d86,202a5e06c5011bf8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:55:44.966 WARN 4486 --- [scanner-13305926-1,f8f1dcc5a5599d86,14cd3f3386b8fe54] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:55:45.103 INFO 4486 --- [alarm-detect-18,f8f1dcc5a5599d86,a7f3bc3805fa469c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058115 ~ 1729058135 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058130, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058130, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058130=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:55:45.341 INFO 4486 --- [ocp-async-8,a1f38e33cba9537c,142570e584a561d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:45.564 INFO 4486 --- [ocp-schedules-20,94b71bfc0c8a05b4,410cf75411461f4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:55:45.569 INFO 4486 --- [ocp-schedules-20,94b71bfc0c8a05b4,410cf75411461f4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:55:45.569 INFO 4486 --- [ocp-schedules-20,94b71bfc0c8a05b4,410cf75411461f4f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:55:45.593 INFO 4486 --- [ocp-schedules-5,21e1ef70f2a2c4d9,72e5b5f108c901f6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:55:45.593 INFO 4486 --- [ocp-schedules-5,21e1ef70f2a2c4d9,72e5b5f108c901f6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:55:46.376 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:46.994 INFO 4486 --- [prometheus-minute-schedule-1,426ea442ababa42b,570d4e5874a3f05d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:47.000 INFO 4486 --- [prometheus-minute-schedule-1,426ea442ababa42b,570d4e5874a3f05d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:47.073 INFO 4486 --- [ocp-schedules-24,49983aee1b92e676,d3db9329136a2ca2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:55:47.206 INFO 4486 --- [ocp-schedules-30,a0164dee6a46951f,1e6ec8b0c3a8f5fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:55:47.232 INFO 4486 --- [ocp-schedules-21,d5ab29ebc89fa322,b424427de8bce6d3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:55:47.342 INFO 4486 --- [ocp-async-4,3ced52dfd9da285b,efb1172ea0537398] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:48.130 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,5163aa11915e3e6b,4924a9d6746594be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5163aa11915e3e6b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:48.130 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,5163aa11915e3e6b,4924a9d6746594be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5163aa11915e3e6b, duration=2 ms] 2024-10-16 13:55:48.758 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:55:49.342 INFO 4486 --- [ocp-async-7,67f98dbd5f4c8eb3,e32abcf3af6169ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:49.378 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:49.994 INFO 4486 --- [prometheus-minute-schedule-1,28c863d6f025b321,404e52564d3e6b3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:50.000 INFO 4486 --- [prometheus-minute-schedule-1,28c863d6f025b321,404e52564d3e6b3c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:50.199 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d0e73d8e1a59e239,78643d8f1e505cc5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d0e73d8e1a59e239, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:50.200 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d0e73d8e1a59e239,78643d8f1e505cc5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d0e73d8e1a59e239, duration=2 ms] 2024-10-16 13:55:50.522 INFO 4486 --- [ocp-schedules-21,ce9ae56c84e22109,5a387efb8874c464] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:55:50.522 INFO 4486 --- [ocp-schedules-21,ce9ae56c84e22109,5a387efb8874c464] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:55:50.523 INFO 4486 --- [ocp-schedules-21,ce9ae56c84e22109,5a387efb8874c464] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:55:50.523 INFO 4486 --- [ocp-schedules-21,ce9ae56c84e22109,5a387efb8874c464] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:55:50.524 INFO 4486 --- [ocp-schedules-21,ce9ae56c84e22109,5a387efb8874c464] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:55:50.524 INFO 4486 --- [ocp-schedules-21,ce9ae56c84e22109,5a387efb8874c464] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:55:51.085 INFO 4486 --- [ocp-schedules-17,4c4457d23f84631e,ef34dcb7a783c623] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:55:51.168 INFO 4486 --- [ocp-schedules-27,f36f9e7ed4f96c53,afcdf62990517bd6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:55:51.168 INFO 4486 --- [ocp-schedules-27,f36f9e7ed4f96c53,afcdf62990517bd6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:55:51.168 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:55:51.168 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:55:51.168 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:55:51.168 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:55:51.168 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:55:51.174 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:55:51.174 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:55:51.174 INFO 4486 --- [ocp-schedules-27,f36f9e7ed4f96c53,afcdf62990517bd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:55:51.174 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:55:51.175 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:55:51.175 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:55:51.176 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:55:51.176 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:55:51.176 INFO 4486 --- [ocp-schedules-27,f36f9e7ed4f96c53,afcdf62990517bd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:55:51.176 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:55:51.176 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:55:51.177 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:55:51.200 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:55:51.201 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:55:51.201 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:55:51.202 INFO 4486 --- [ocp-schedules-27,f36f9e7ed4f96c53,afcdf62990517bd6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:55:51.202 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:55:51.203 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:55:51.209 INFO 4486 --- [ocp-schedules-27,f36f9e7ed4f96c53,afcdf62990517bd6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:55:51.342 INFO 4486 --- [ocp-async-8,2e7ead9fd50f1bda,a9b4d6ec7cc56b48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:51.739 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,34ca6b4e220deddf,465e9869a5e00a1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=34ca6b4e220deddf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:51.739 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,34ca6b4e220deddf,465e9869a5e00a1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=34ca6b4e220deddf, duration=1 ms] 2024-10-16 13:55:52.017 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b4b29c8d328e47f8,bcdb11b5f1c162b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b4b29c8d328e47f8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:52.017 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b4b29c8d328e47f8,bcdb11b5f1c162b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b4b29c8d328e47f8, duration=1 ms] 2024-10-16 13:55:52.379 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:52.413 INFO 4486 --- [ocp-schedules-6,b08d5a358ea3515e,cf36768011f0e4fe] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:55:52.994 INFO 4486 --- [prometheus-minute-schedule-1,1434b60e725f67a6,40b0b596a4113506] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:53.001 INFO 4486 --- [prometheus-minute-schedule-1,1434b60e725f67a6,40b0b596a4113506] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:53.020 INFO 4486 --- [ocp-async-2,a1033adf78cb763d,f0c74389d1595216] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:55:53.342 INFO 4486 --- [ocp-async-4,4afd2de7fd3e6089,929dd011f92672f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:55.197 INFO 4486 --- [ocp-schedules-4,b0c42e44f245180a,92099219a3c0ee2d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:55:55.237 WARN 4486 --- [scanner-13306173-1,b0c42e44f245180a,3e71d1c5d423b9d0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:55:55.315 INFO 4486 --- [alarm-detect-18,b0c42e44f245180a,6444e913e760e6c0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058130 ~ 1729058145 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058145, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058145, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058145=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:55:55.344 INFO 4486 --- [ocp-async-3,8c7cfea52ba9b35f,aa69d1a1ab4e8619] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:55.380 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:55.994 INFO 4486 --- [prometheus-minute-schedule-1,c2f9749cf3210947,b62a3e0c6312ac05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:55.999 INFO 4486 --- [prometheus-minute-schedule-1,c2f9749cf3210947,b62a3e0c6312ac05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:57.344 INFO 4486 --- [ocp-async-8,d21ab9b39b8ad03e,bb1a6d6ecf14b962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:55:58.139 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f1cd55718a775703,63ce9d315a311e1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f1cd55718a775703, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:55:58.140 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f1cd55718a775703,63ce9d315a311e1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f1cd55718a775703, duration=2 ms] 2024-10-16 13:55:58.381 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:55:58.994 INFO 4486 --- [prometheus-minute-schedule-1,9ceeede4f815e460,0c47822c9c4890ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058160, interval=60 2024-10-16 13:55:59.001 INFO 4486 --- [prometheus-minute-schedule-1,9ceeede4f815e460,0c47822c9c4890ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:55:59.183 INFO 4486 --- [ocp-schedules-29,76d1c7e5220e0c23,5a5189f291cd4f8c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:55:59.345 INFO 4486 --- [ocp-async-10,a771f0519db9d094,657f6ea5904c80e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:00.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4b17c0068cb5c023,a6741fd2e4e096c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4b17c0068cb5c023, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:00.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4b17c0068cb5c023,a6741fd2e4e096c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4b17c0068cb5c023, duration=2 ms] 2024-10-16 13:56:01.091 INFO 4486 --- [ocp-schedules-24,3ce2dd77b15fa5c8,75865050184b8e80] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:56:01.345 INFO 4486 --- [ocp-async-7,05822ec15af516e7,7223293d467e1636] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:01.383 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:01.738 INFO 4486 --- [ocp-schedules-16,e5e826675c61b41f,8efacf683307f4d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:56:01.738 INFO 4486 --- [ocp-schedules-16,df5b23c847d14e7a,35351b2c2ca8531d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:56:01.740 INFO 4486 --- [ocp-schedules-16,df5b23c847d14e7a,35351b2c2ca8531d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:01.746 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1555666aee787f66,aa9f450675eb19e7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1555666aee787f66, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:01.746 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,1555666aee787f66,aa9f450675eb19e7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1555666aee787f66, duration=3 ms] 2024-10-16 13:56:01.774 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.775 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:01.793 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.794 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:01.811 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.812 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:01.812 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:01.812 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.813 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:56:01.847 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:01.847 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.848 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:01.849 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:56:01.849 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.852 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:56:01.854 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:01.864 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:01.865 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:01.875 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:01.883 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:01.883 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.884 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:01.885 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:56:01.891 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:56:01.896 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:01.899 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:01.925 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:01.927 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.927 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:56:01.930 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:56:01.994 INFO 4486 --- [prometheus-minute-schedule-1,5256730343eea1e0,f6bf365400ae3d2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:01.995 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:01.997 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:01.998 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:02.003 INFO 4486 --- [prometheus-minute-schedule-1,5256730343eea1e0,f6bf365400ae3d2e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:02.003 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:02.006 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:56:02.008 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:56:02.008 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:56:02.011 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:02.064 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:56:02.065 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,92a9d247d26c2707,434da0bc3a36fcf9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=92a9d247d26c2707, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:02.065 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,92a9d247d26c2707,434da0bc3a36fcf9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=92a9d247d26c2707, duration=1 ms] 2024-10-16 13:56:02.075 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:02.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:02.077 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:56:02.086 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:56:02.086 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:56:02.094 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:02.094 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:02.122 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:02.122 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:02.148 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:02.148 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:02.175 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:02.175 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:02.203 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:02.203 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:02.233 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:02.260 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:02.261 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:02.262 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:02.267 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:56:02.272 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:56:02.277 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:56:02.279 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:56:02.303 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:56:02.372 INFO 4486 --- [ocp-schedules-25,b8215b9dc212e441,3f32a0d462171054] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:56:02.372+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:56:02.377 INFO 4486 --- [ocp-schedules-25,b8215b9dc212e441,3f32a0d462171054] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:56:02.387 INFO 4486 --- [ocp-schedules-25,b8215b9dc212e441,3f32a0d462171054] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 13:56:02.398 INFO 4486 --- [ocp-schedules-25,b8215b9dc212e441,3f32a0d462171054] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:56:02.447 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:56:02.472 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:02.473 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:02.474 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:56:02.503 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:56:02.503 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:56:03.021 INFO 4486 --- [ocp-async-4,a3ac396d17bc9aee,4fd5005fa43b51ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:56:03.345 INFO 4486 --- [ocp-async-2,6c99f7387840dfaf,4955d8719cb2cac6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:04.383 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:04.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d52eb5b71d8ee9a9,befaf3cf2ec86ead] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=d52eb5b71d8ee9a9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:56:04.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,d52eb5b71d8ee9a9,befaf3cf2ec86ead] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=d52eb5b71d8ee9a9, duration=1 ms] 2024-10-16 13:56:04.994 INFO 4486 --- [prometheus-minute-schedule-1,ab3b7973b12d838d,2baaf6362b83cea9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:05.001 INFO 4486 --- [prometheus-minute-schedule-1,ab3b7973b12d838d,2baaf6362b83cea9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:05.345 INFO 4486 --- [ocp-async-5,e9f26c148670973f,51c042689efd5e9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:05.400 INFO 4486 --- [ocp-schedules-7,ffd0704ece982448,8e45dfd77c4ff37d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:56:05.446 WARN 4486 --- [scanner-13306558-1,ffd0704ece982448,446ab494dc4a4c8f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:56:07.066 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:56:07.066 INFO 4486 --- [ocp-schedules-29,2b852eb06d347337,cbdffa9077167009] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:56:07.069 INFO 4486 --- [ocp-schedules-29,2b852eb06d347337,cbdffa9077167009] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:07.081 INFO 4486 --- [ocp-schedules-24,3ac3829cae9ee6e3,3b5a186b89316e46] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:56:07.083 INFO 4486 --- [ocp-schedules-24,3ac3829cae9ee6e3,3b5a186b89316e46] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:56:07.089 INFO 4486 --- [ocp-schedules-2,7ceaa55a94f5872d,de390b8c4a287214] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:56:07.090 INFO 4486 --- [ocp-schedules-26,2c51f6f9622c4e6c,162c7c896e679ad6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:56:07.090 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:56:07.090 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:56:07.090 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:56:07.091 INFO 4486 --- [host-check-clockdiff1,2c51f6f9622c4e6c,d805637a0539a824] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:56:07.091 INFO 4486 --- [host-check-clockdiff2,2c51f6f9622c4e6c,9589a42117aa2291] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:56:07.091 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:56:07.091 INFO 4486 --- [ocp-schedules-5,94ff37259e690dc7,cb1f61926bab7600] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:56:07.092 INFO 4486 --- [host-check-clockdiff3,2c51f6f9622c4e6c,e4cb8446264fea02] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:56:07.092 INFO 4486 --- [host-check-agent-status1,94ff37259e690dc7,ac32d45dc70b8ff6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:56:07.092 INFO 4486 --- [host-check-clockdiff4,2c51f6f9622c4e6c,fba577b75e6e76fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:56:07.092 INFO 4486 --- [host-check-agent-status3,94ff37259e690dc7,a6e510ac973c9d96] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:56:07.092 INFO 4486 --- [host-check-agent-status2,94ff37259e690dc7,7184eb265ec9d0f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:56:07.093 INFO 4486 --- [host-check-agent-status4,94ff37259e690dc7,96e2869a98fb2296] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:56:07.097 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:07.098 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:56:07.101 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:56:07.102 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:56:07.108 INFO 4486 --- [host-check-agent-status4,94ff37259e690dc7,96e2869a98fb2296] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:56:07.108 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:07.108 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:56:07.120 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:56:07.120 INFO 4486 --- [host-check-agent-status4,94ff37259e690dc7,96e2869a98fb2296] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:56:07.127 INFO 4486 --- [host-check-agent-status2,94ff37259e690dc7,7184eb265ec9d0f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:56:07.131 INFO 4486 --- [host-check-agent-status3,94ff37259e690dc7,a6e510ac973c9d96] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:56:07.131 INFO 4486 --- [host-check-agent-status1,94ff37259e690dc7,ac32d45dc70b8ff6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:07.131 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:56:07.134 INFO 4486 --- [host-check-clockdiff1,2c51f6f9622c4e6c,d805637a0539a824] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058167 0 0 , error= 2024-10-16 13:56:07.134 INFO 4486 --- [host-check-agent-status3,94ff37259e690dc7,a6e510ac973c9d96] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:56:07.134 INFO 4486 --- [host-check-clockdiff3,2c51f6f9622c4e6c,e4cb8446264fea02] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058167 5 5 , error= 2024-10-16 13:56:07.136 INFO 4486 --- [host-check-agent-status2,94ff37259e690dc7,7184eb265ec9d0f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:56:07.134 INFO 4486 --- [host-check-clockdiff2,2c51f6f9622c4e6c,9589a42117aa2291] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058167 5 5 , error= 2024-10-16 13:56:07.137 INFO 4486 --- [host-check-agent-status1,94ff37259e690dc7,ac32d45dc70b8ff6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:07.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:56:07.159 INFO 4486 --- [host-check-agent-status4,94ff37259e690dc7,96e2869a98fb2296] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:56:07.160 INFO 4486 --- [host-check-agent-status3,94ff37259e690dc7,a6e510ac973c9d96] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:56:07.161 INFO 4486 --- [host-check-agent-status1,94ff37259e690dc7,ac32d45dc70b8ff6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:56:07.161 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:56:07.161 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:56:07.162 INFO 4486 --- [host-check-agent-status2,94ff37259e690dc7,7184eb265ec9d0f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:56:07.163 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:56:07.163 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:56:07.168023809+08:00 2024-10-16 13:56:07.163 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status1,94ff37259e690dc7,ac32d45dc70b8ff6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status4,94ff37259e690dc7,96e2869a98fb2296] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status4,94ff37259e690dc7,96e2869a98fb2296] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status3,94ff37259e690dc7,a6e510ac973c9d96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status1,94ff37259e690dc7,ac32d45dc70b8ff6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status4,94ff37259e690dc7,96e2869a98fb2296] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status3,94ff37259e690dc7,a6e510ac973c9d96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status1,94ff37259e690dc7,ac32d45dc70b8ff6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-agent-status3,94ff37259e690dc7,a6e510ac973c9d96] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:56:07.164172274+08:00 2024-10-16 13:56:07.168 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:56:07.164+08:00 2024-10-16 13:56:07.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.169 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:56:07.168+08:00 2024-10-16 13:56:07.170 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:07.170 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:56:07.174572328+08:00 2024-10-16 13:56:07.170 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:56:07.174247041+08:00 2024-10-16 13:56:07.171 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:56:07.170+08:00 2024-10-16 13:56:07.171 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:56:07.170+08:00 2024-10-16 13:56:07.171 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:56:07.164+08:00 and status AVAILABLE 2024-10-16 13:56:07.172 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:56:07.168+08:00 and status AVAILABLE 2024-10-16 13:56:07.172 INFO 4486 --- [host-check-agent-status2,94ff37259e690dc7,7184eb265ec9d0f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:56:07.173 INFO 4486 --- [host-check-agent-status2,94ff37259e690dc7,7184eb265ec9d0f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:56:07.173 INFO 4486 --- [host-check-agent-status2,94ff37259e690dc7,7184eb265ec9d0f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:56:07.173 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:56:07.170+08:00 and status AVAILABLE 2024-10-16 13:56:07.173 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:56:07.170+08:00 and status AVAILABLE 2024-10-16 13:56:07.174 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:56:07.175 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:56:07.175 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:56:07.175 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:56:07.176 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:56:07.178 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.183 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:56:07.184 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:07.184 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:56:07.185 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:56:07.186 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:56:07.186 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:07.186 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:56:07.186 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:56:07.187 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:56:07.191 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.192 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:07.201 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.202 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:07.207 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:07.209 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:07.209 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:07.211 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:07.214 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.215 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:07.219 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.220 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:56:07.232 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:07.233 INFO 4486 --- [ocp-schedules-9,e91c50136ad9dead,60c7fa3e6478c743] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:56:07.235 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:56:07.242 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:56:07.245 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:56:07.247 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:56:07.255 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:56:07.261 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:56:07.264 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:07.268 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:56:07.276 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:07.276 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:07.279 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:56:07.292 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:56:07.296 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:07.302 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:56:07.311 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:56:07.313 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:56:07.318 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:07.325 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:56:07.326 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:56:07.343 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:07.345 INFO 4486 --- [ocp-async-9,5476e00f1b8f099f,5440398dcf1c0911] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:07.347 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:56:07.349 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:07.354 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:56:07.359 INFO 4486 --- [host-check-host-status3,7ceaa55a94f5872d,32a87050d1e726f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:56:07.365 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:56:07.375 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:56:07.383 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:07.384 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:07.386 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:56:07.388 INFO 4486 --- [host-check-host-status4,7ceaa55a94f5872d,77f276766699599e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:56:07.399 INFO 4486 --- [host-check-host-status2,7ceaa55a94f5872d,937114ddf1f313dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:56:07.436 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:56:07.446 INFO 4486 --- [host-check-host-status1,7ceaa55a94f5872d,b106acf009fad15a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:56:07.994 INFO 4486 --- [prometheus-minute-schedule-1,e41fd8909b1bedd5,07ed6d30545ba61e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:08.000 INFO 4486 --- [prometheus-minute-schedule-1,e41fd8909b1bedd5,07ed6d30545ba61e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:08.058 INFO 4486 --- [host-check-clockdiff4,2c51f6f9622c4e6c,fba577b75e6e76fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058168 4 4 , error= 2024-10-16 13:56:08.144 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,70a3cc6c0e898825,769d227e84ccdda0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=70a3cc6c0e898825, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:08.144 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,70a3cc6c0e898825,769d227e84ccdda0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=70a3cc6c0e898825, duration=1 ms] 2024-10-16 13:56:09.346 INFO 4486 --- [ocp-async-6,957a720c554fdaf6,6362c381e3d9ca22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:10.006 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:56:10.007 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:56:10.073 INFO 4486 --- [Check obproxy2,7a58fc732af7de0d,c27d526fa9e719b9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:56:10.073 INFO 4486 --- [Check obproxy2,7a58fc732af7de0d,c27d526fa9e719b9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:10.074 INFO 4486 --- [Check obproxy1,11c44e9c942df509,1bf3cf986febf097] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:56:10.074 INFO 4486 --- [Check obproxy1,11c44e9c942df509,1bf3cf986febf097] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:10.078 INFO 4486 --- [Collect obproxy1,1f167611a367852a,0a6cf71c8276b111] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:56:10.078 INFO 4486 --- [Collect obproxy1,1f167611a367852a,0a6cf71c8276b111] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:10.078 INFO 4486 --- [Collect obproxy2,1c970901f5557aef,33422a7a0369d2fe] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:56:10.078 INFO 4486 --- [Collect obproxy2,1c970901f5557aef,33422a7a0369d2fe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:10.098 INFO 4486 --- [Check obproxy2,7a58fc732af7de0d,c27d526fa9e719b9] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:56:10.098 INFO 4486 --- [Check obproxy2,7a58fc732af7de0d,c27d526fa9e719b9] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:56:10.100 INFO 4486 --- [Collect obproxy2,1c970901f5557aef,33422a7a0369d2fe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:56:10.101 INFO 4486 --- [Check obproxy1,11c44e9c942df509,1bf3cf986febf097] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:56:10.101 INFO 4486 --- [Check obproxy1,11c44e9c942df509,1bf3cf986febf097] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:56:10.104 INFO 4486 --- [Collect obproxy1,1f167611a367852a,0a6cf71c8276b111] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:56:10.105 INFO 4486 --- [Collect obproxy2,1c970901f5557aef,33422a7a0369d2fe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:56:10.109 INFO 4486 --- [Collect obproxy1,1f167611a367852a,0a6cf71c8276b111] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:56:10.111 INFO 4486 --- [Collect obproxy2,1c970901f5557aef,33422a7a0369d2fe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:56:10.115 INFO 4486 --- [Collect obproxy1,1f167611a367852a,0a6cf71c8276b111] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:56:10.209 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2cee980005ea4544,ab6f77146b896230] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2cee980005ea4544, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:10.209 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2cee980005ea4544,ab6f77146b896230] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2cee980005ea4544, duration=1 ms] 2024-10-16 13:56:10.385 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:56:10.994 INFO 4486 --- [prometheus-minute-schedule-1,7de31fe15e936a4c,b4d187ccb828de5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:11.000 INFO 4486 --- [prometheus-minute-schedule-1,7de31fe15e936a4c,b4d187ccb828de5d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:56:11.096 INFO 4486 --- [ocp-schedules-9,231590dfcca07baf,9f330c750c4a4976] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:56:11.346 INFO 4486 --- [ocp-async-1,b2b0e845a24cebae,b3005f039d10ee30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:11.752 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c80ccb9a1af20f0f,d00fa53d3fdb7784] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c80ccb9a1af20f0f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:11.752 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c80ccb9a1af20f0f,d00fa53d3fdb7784] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c80ccb9a1af20f0f, duration=1 ms] 2024-10-16 13:56:12.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5a7a80897c3c2a75,4ff8352446c0777d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5a7a80897c3c2a75, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:12.069 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5a7a80897c3c2a75,4ff8352446c0777d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5a7a80897c3c2a75, duration=1 ms] 2024-10-16 13:56:12.941 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:56:12.941 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:56:12.950 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:56:12.950 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:56:12.950 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:56:12.951 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:56:12.958 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:56:12.958 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:56:12.958 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:56:12.958 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:56:13.013 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:56:13.021 INFO 4486 --- [ocp-async-8,efa857bf53b3041a,8fd08775d79227cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:56:13.022 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:56:13.023 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:56:13.023 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:56:13.023 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:56:13.023 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:56:13.031 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:56:13.346 INFO 4486 --- [ocp-async-9,a3f6924e93aa900f,a7d92daf350238db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:13.386 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:13.595 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:56:13.644 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,510f83984015503e,8db56361a38cc1a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=510f83984015503e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:56:13.645 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,510f83984015503e,8db56361a38cc1a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=510f83984015503e, duration=3 ms] 2024-10-16 13:56:13.994 INFO 4486 --- [prometheus-minute-schedule-1,37c854d40a530a8e,f8278b42f8fbdae4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:14.001 INFO 4486 --- [prometheus-minute-schedule-1,37c854d40a530a8e,f8278b42f8fbdae4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:14.216 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.257623167S 2024-10-16 13:56:14.216 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:56:14.284 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:56:14.284 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:56:14.310 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:56:14.322 ERROR 4486 --- [metric-archive-second-25,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '587073990944165888-1729058100' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:56:14.321106] [YB420AC531B8-00060B7DC0C87EE6-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '587073990944165888-1729058100' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:56:14.321106] [YB420AC531B8-00060B7DC0C87EE6-0-0] 2024-10-16 13:56:14.339 ERROR 4486 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1694585) Duplicate entry '669741696503320576-1729058100' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:56:14.322907] [YB420AC531B8-00060B7DA1ABEA15-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1694585) Duplicate entry '669741696503320576-1729058100' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:56:14.322907] [YB420AC531B8-00060B7DA1ABEA15-0-0] 2024-10-16 13:56:14.339 ERROR 4486 --- [metric-archive-second-23,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=835727) Duplicate entry '669741699544190976-1729058100' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:56:14.323460] [YB420AC531B8-00060B7DB41968D7-0-0]; nested exception is java.sql.BatchUpdateException: (conn=835727) Duplicate entry '669741699544190976-1729058100' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:56:14.323460] [YB420AC531B8-00060B7DB41968D7-0-0] 2024-10-16 13:56:14.342 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058100, nodeCount=1000, affectRows=1000 2024-10-16 13:56:14.397 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:56:14.398 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:56:14.398 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:56:14.398 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:56:14.398 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:56:14.398 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:56:14.407 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:56:14.438 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058100, nodeCount=169, affectRows=169 2024-10-16 13:56:14.439 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058100, nodeCount=168, validCount=168 2024-10-16 13:56:14.465 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058100, nodeCount=211, affectRows=211 2024-10-16 13:56:14.465 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058100, nodeCount=211, validCount=211 2024-10-16 13:56:14.527 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] o.s.boot.SpringApplication : Started application in 1.577 seconds (JVM running for 79077.578) 2024-10-16 13:56:14.527 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.5774386S 2024-10-16 13:56:14.934 INFO 4486 --- [ocp-schedules-21,bce93bdb56af722c,264359812f075f11] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:56:15.117 INFO 4486 --- [metric-archive-async-19,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058100, nodeCount=176, affectRows=176 2024-10-16 13:56:15.117 INFO 4486 --- [metric-archive-async-12,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058100, nodeCount=174, affectRows=174 2024-10-16 13:56:15.118 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058100, nodeCount=127, affectRows=127 2024-10-16 13:56:15.301 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058100, nodeCount=127, validCount=127 2024-10-16 13:56:15.346 INFO 4486 --- [ocp-async-6,e3867711166779d0,62f15ff99a33da46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:15.360 INFO 4486 --- [metric-archive-async-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058100, nodeCount=176, validCount=176 2024-10-16 13:56:15.365 INFO 4486 --- [metric-archive-async-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058100, nodeCount=174, validCount=174 2024-10-16 13:56:15.537 INFO 4486 --- [ocp-schedules-7,8f92086ff168c319,a0bf4a3c5c621e1e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:56:15.566 WARN 4486 --- [scanner-13307583-1,8f92086ff168c319,79b0695c6a12c822] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:56:15.617 INFO 4486 --- [alarm-detect-10,8f92086ff168c319,2ccbbab87bbf0bb7] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058145 ~ 1729058165 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058160, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058160, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058160=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:56:16.387 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:16.994 INFO 4486 --- [prometheus-minute-schedule-1,038eeda4e32a02f1,de5b64e2981e32e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:17.000 INFO 4486 --- [prometheus-minute-schedule-1,038eeda4e32a02f1,de5b64e2981e32e4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:17.071 INFO 4486 --- [ocp-schedules-29,2b852eb06d347337,cbdffa9077167009] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:56:17.175 INFO 4486 --- [ocp-schedules-24,3ac3829cae9ee6e3,3b5a186b89316e46] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:56:17.246 INFO 4486 --- [ocp-schedules-12,869e492f15d8a514,3eafac5e6e683881] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:56:17.346 INFO 4486 --- [ocp-async-1,dd36de0a46851380,32324a7943a62279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:18.149 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc24a09797fb8541,1c80987ee1bc90af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cc24a09797fb8541, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:18.149 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,cc24a09797fb8541,1c80987ee1bc90af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cc24a09797fb8541, duration=2 ms] 2024-10-16 13:56:19.189 INFO 4486 --- [ocp-schedules-21,0fbcaa5e0c757432,5d557e7300b2694f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:56:19.347 INFO 4486 --- [ocp-async-4,d3fc9ac1f3840b09,5c225f7304df4eee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:19.388 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:19.994 INFO 4486 --- [prometheus-minute-schedule-1,7bc5792ac2d63466,5bca50c0e133c68e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:19.999 INFO 4486 --- [prometheus-minute-schedule-1,7bc5792ac2d63466,5bca50c0e133c68e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:20.218 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,012f87aaf0104b9b,7dda35340e972821] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=012f87aaf0104b9b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:20.218 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,012f87aaf0104b9b,7dda35340e972821] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=012f87aaf0104b9b, duration=1 ms] 2024-10-16 13:56:20.813 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:56:20.996 INFO 4486 --- [prometheus-cache-monitor1,4e62b4b7bdd6925d,161fdd5fe666c25f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8412} 2024-10-16 13:56:20.996 INFO 4486 --- [prometheus-cache-monitor1,4e62b4b7bdd6925d,161fdd5fe666c25f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:56:21.102 INFO 4486 --- [ocp-schedules-11,001e2c277ac43c94,895cfa214f4823de] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:56:21.347 INFO 4486 --- [ocp-async-3,c049cafb0f5504d8,b0dd46fe4c773f27] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:21.758 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,14c882d03741c634,e7e504ba85293903] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=14c882d03741c634, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:21.759 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,14c882d03741c634,e7e504ba85293903] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=14c882d03741c634, duration=2 ms] 2024-10-16 13:56:22.074 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8ebd8a9be5605461,02158e878a17b707] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8ebd8a9be5605461, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:22.074 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8ebd8a9be5605461,02158e878a17b707] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8ebd8a9be5605461, duration=1 ms] 2024-10-16 13:56:22.389 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:22.447 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4b24752ee9303806,afefb4233d656926] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=4b24752ee9303806, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:56:22.447 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4b24752ee9303806,afefb4233d656926] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=4b24752ee9303806, duration=1 ms] 2024-10-16 13:56:22.994 INFO 4486 --- [prometheus-minute-schedule-1,b6c1a31a0e537ea2,08f1a93effc9e383] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:23.000 INFO 4486 --- [prometheus-minute-schedule-1,b6c1a31a0e537ea2,08f1a93effc9e383] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:23.021 INFO 4486 --- [ocp-async-7,f5ce38ebb54615ac,346a8f4eebd151fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:56:23.347 INFO 4486 --- [ocp-async-1,9fb3b6f005e27410,e65c50809af6e07f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:23.760 INFO 4486 --- [ocp-schedules-5,f9f9cb8b8a8f64b9,387e91ec3c6d3061] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:56:25.053 INFO 4486 --- [ocp-schedules-17,697f9575f86edb0f,39ac5238131ec782] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:56:25.347 INFO 4486 --- [ocp-async-4,ba2ac05a6f65bbad,fd091f48563f83f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:25.390 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:25.702 INFO 4486 --- [ocp-schedules-20,283f28c6330920b5,25bd9d3586412be1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:56:25.732 WARN 4486 --- [scanner-13307899-1,283f28c6330920b5,759a8c79b8c6a0b5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:56:25.816 INFO 4486 --- [alarm-detect-20,283f28c6330920b5,ecd391902e8c5e03] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058160 ~ 1729058175 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058175, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058175, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058175=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:56:25.994 INFO 4486 --- [prometheus-minute-schedule-1,2b3f5675595fb656,6e6d5848e526a90f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:26.001 INFO 4486 --- [prometheus-minute-schedule-1,2b3f5675595fb656,6e6d5848e526a90f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:26.387 INFO 4486 --- [ocp-schedules-14,31120e640af27338,ef0a34bd84119458] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:56:26.387 INFO 4486 --- [ocp-schedules-14,31120e640af27338,ef0a34bd84119458] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:56:26.418 INFO 4486 --- [Collect all tenants major compactions2,31120e640af27338,2b75b7f6cfca8869] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:56:26.418 INFO 4486 --- [Collect all tenants major compactions1,31120e640af27338,2df90e0d2155542b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:56:26.418 INFO 4486 --- [Collect all tenants major compactions3,31120e640af27338,b7e0d01b67de0b08] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:56:26.419 INFO 4486 --- [Collect all tenants major compactions4,31120e640af27338,171c8c79855ff191] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:56:26.419 INFO 4486 --- [Collect all tenants major compactions5,31120e640af27338,c974b17e3353ce9e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:56:26.438 INFO 4486 --- [Collect all tenants major compactions2,31120e640af27338,2b75b7f6cfca8869] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:26.440 INFO 4486 --- [Collect all tenants major compactions1,31120e640af27338,2df90e0d2155542b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:26.440 INFO 4486 --- [Collect all tenants major compactions2,31120e640af27338,2b75b7f6cfca8869] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:56:26.441 INFO 4486 --- [Collect all tenants major compactions1,31120e640af27338,2df90e0d2155542b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:56:26.442 INFO 4486 --- [Collect all tenants major compactions3,31120e640af27338,b7e0d01b67de0b08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:26.443 INFO 4486 --- [Collect all tenants major compactions2,31120e640af27338,2b75b7f6cfca8869] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:56:26.444 INFO 4486 --- [Collect all tenants major compactions3,31120e640af27338,b7e0d01b67de0b08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:56:26.444 INFO 4486 --- [Collect all tenants major compactions1,31120e640af27338,2df90e0d2155542b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:56:26.447 INFO 4486 --- [Collect all tenants major compactions2,31120e640af27338,2b75b7f6cfca8869] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:56:26.447 INFO 4486 --- [Collect all tenants major compactions3,31120e640af27338,b7e0d01b67de0b08] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:56:26.447 INFO 4486 --- [Collect all tenants major compactions3,31120e640af27338,b7e0d01b67de0b08] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:56:26.449 INFO 4486 --- [Collect all tenants major compactions4,31120e640af27338,171c8c79855ff191] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:26.449 INFO 4486 --- [Collect all tenants major compactions5,31120e640af27338,c974b17e3353ce9e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:26.450 INFO 4486 --- [Collect all tenants major compactions1,31120e640af27338,2df90e0d2155542b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:56:26.451 INFO 4486 --- [Collect all tenants major compactions4,31120e640af27338,171c8c79855ff191] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:56:26.452 INFO 4486 --- [Collect all tenants major compactions5,31120e640af27338,c974b17e3353ce9e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:56:26.454 INFO 4486 --- [Collect all tenants major compactions4,31120e640af27338,171c8c79855ff191] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:56:26.454 INFO 4486 --- [Collect all tenants major compactions4,31120e640af27338,171c8c79855ff191] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:56:26.457 INFO 4486 --- [Collect all tenants major compactions5,31120e640af27338,c974b17e3353ce9e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:56:26.457 INFO 4486 --- [Collect all tenants major compactions5,31120e640af27338,c974b17e3353ce9e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:56:26.459 INFO 4486 --- [ocp-schedules-14,31120e640af27338,ef0a34bd84119458] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:56:27.347 INFO 4486 --- [ocp-async-7,4316fe9b928ffef0,9f33c1ef0d43ab73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:27.690 INFO 4486 --- [ocp-schedules-23,6524d9ebfdf2551d,9cea36684108958f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:56:27.690+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:56:27.692 INFO 4486 --- [ocp-schedules-23,6524d9ebfdf2551d,9cea36684108958f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:56:27.692 INFO 4486 --- [ocp-schedules-23,6524d9ebfdf2551d,9cea36684108958f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:56:27.693 INFO 4486 --- [ocp-schedules-23,6524d9ebfdf2551d,9cea36684108958f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:56:28.054 INFO 4486 --- [ocp-schedules-1,209879f0d6f1ac0b,b9d5ec8b4b6c9754] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:56:28.056 INFO 4486 --- [ocp-schedules-1,209879f0d6f1ac0b,b9d5ec8b4b6c9754] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:28.091 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:56:28.092 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.095 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:56:28.102 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:56:28.103 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.107 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.107 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:56:28.107 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:56:28.114 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.115 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:56:28.122 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.122 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:56:28.127 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.128 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:56:28.131 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.131 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:56:28.134 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:56:28.135 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.139 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:56:28.140 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.140 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.140 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:28.141 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:56:28.149 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.149 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:28.150 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.151 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:56:28.152 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.153 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:28.156 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4743499c598c94d9,6131f5ad36e2d5c0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4743499c598c94d9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:28.157 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4743499c598c94d9,6131f5ad36e2d5c0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4743499c598c94d9, duration=3 ms] 2024-10-16 13:56:28.158 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.159 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:56:28.163 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.164 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:28.164 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.165 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:56:28.166 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.167 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:56:28.175 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.176 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:56:28.180 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.181 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:56:28.186 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.186 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:56:28.187 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.188 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:56:28.192 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.193 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.194 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.195 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.195 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.196 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:56:28.196 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.197 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.198 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.199 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.199 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.200 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.201 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.202 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.202 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.203 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.203 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.204 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.205 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.205 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.205 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.206 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:56:28.206 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.207 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.207 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.208 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.208 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.208 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.208 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:28.209 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.210 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:56:28.215 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.216 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:56:28.217 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.217 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.218 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:28.219 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.221 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.223 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.224 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.226 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.227 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.228 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:28.229 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.232 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.233 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.233 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:28.234 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.234 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:56:28.235 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.236 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.241 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.243 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.245 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.247 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:56:28.251 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:56:28.251 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:56:28.252 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.253 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.256 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:56:28.259 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:56:28.260 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.261 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.264 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.265 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.268 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.269 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.272 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.273 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.276 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.277 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.280 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.281 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.284 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.286 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:56:28.289 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.290 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:56:28.290 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:28.292 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.293 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.293 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:56:28.294 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:56:28.297 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:56:28.300 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:56:28.306 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.306 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:56:28.306 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.306 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:56:28.306 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.306 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:56:28.310 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:56:28.310 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:56:28.356 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:28.358 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:28.358 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:56:28.369 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.369 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:56:28.369 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.369 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:56:28.369 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:56:28.369 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:56:28.372 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:56:28.372 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:56:28.390 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:28.994 INFO 4486 --- [prometheus-minute-schedule-1,896b5b3b392e7999,5189265c9bb818db] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:28.999 INFO 4486 --- [prometheus-minute-schedule-1,896b5b3b392e7999,5189265c9bb818db] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:29.348 INFO 4486 --- [ocp-async-2,47da4c7ad66cd6ff,097306898b4e0d3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:30.103 INFO 4486 --- [prometheus-second-schedule-1,96640bac5efd9574,22c33177ef0f67e3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:56:30.225 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e7877ad93882d26c,7eb5154cf09be8d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e7877ad93882d26c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:30.225 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e7877ad93882d26c,7eb5154cf09be8d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e7877ad93882d26c, duration=1 ms] 2024-10-16 13:56:31.107 INFO 4486 --- [ocp-schedules-24,384426b2b78e6e04,599ba40f819c3672] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:56:31.348 INFO 4486 --- [ocp-async-10,10933d875c622ba8,53d1a2827b50428b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:31.391 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:31.395 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-10-16 13:56:31.398 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=observer, size=5 2024-10-16 13:56:31.400 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=obproxy, size=3 2024-10-16 13:56:31.401 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=observer, size=5 2024-10-16 13:56:31.402 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=obproxy, size=3 2024-10-16 13:56:31.404 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=observer, size=5 2024-10-16 13:56:31.405 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=obproxy, size=3 2024-10-16 13:56:31.406 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=observer, size=5 2024-10-16 13:56:31.407 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=obproxy, size=3 2024-10-16 13:56:31.408 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=observer, size=5 2024-10-16 13:56:31.409 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=obproxy, size=3 2024-10-16 13:56:31.411 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=observer, size=5 2024-10-16 13:56:31.412 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=obproxy, size=3 2024-10-16 13:56:31.413 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=observer, size=5 2024-10-16 13:56:31.415 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=obproxy, size=3 2024-10-16 13:56:31.415 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-10-16 13:56:31.415 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=40024 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703041150362652672 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703041150358458368 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703041150354264064 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703041150396207107 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703041150396207106 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703041150396207105 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703041150396207104 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703041150396207109 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703041150396207108 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703041150392012801 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703041150392012800 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703041150387818498 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703041150387818497 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703041150387818496 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703041150383624192 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703041150379429890 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703041150379429889 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703041150379429888 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703041150375235585 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703041150375235584 2024-10-16 13:56:31.706 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703041150371041280 2024-10-16 13:56:31.761 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703041150366846977 2024-10-16 13:56:31.761 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703041150366846976 2024-10-16 13:56:31.765 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,104ed6d63e37d7be,27089634850d1a9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=104ed6d63e37d7be, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:31.766 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,104ed6d63e37d7be,27089634850d1a9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=104ed6d63e37d7be, duration=1 ms] 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703041150484287488 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703041150475898880 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703041150471704576 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703041150467510272 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703041150429761537 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703041150429761536 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703041150425567234 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703041150425567233 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703041150425567232 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703041150421372929 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703041150421372928 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703041150417178627 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703041150417178626 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703041150417178625 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703041150417178624 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703041150417178628 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703041150412984323 2024-10-16 13:56:31.922 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703041150412984322 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703041150412984321 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703041150412984320 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703041150408790016 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703041150404595715 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703041150404595714 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703041150404595713 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703041150404595712 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703041150404595717 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703041150404595716 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703041150463315968 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703041150459121664 2024-10-16 13:56:31.923 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703041150450733056 2024-10-16 13:56:31.982 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703041150442344448 2024-10-16 13:56:31.982 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703041150438150144 2024-10-16 13:56:31.982 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703041150433955840 2024-10-16 13:56:31.994 INFO 4486 --- [prometheus-minute-schedule-1,c04bd2e0de2e2212,780784d7cb78379f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:31.999 INFO 4486 --- [prometheus-minute-schedule-1,c04bd2e0de2e2212,780784d7cb78379f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:32.078 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,78bba4ba1d0dc6b6,a4f546d870cf76fb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=78bba4ba1d0dc6b6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:32.078 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,78bba4ba1d0dc6b6,a4f546d870cf76fb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=78bba4ba1d0dc6b6, duration=1 ms] 2024-10-16 13:56:32.504 INFO 4486 --- [ocp-schedules-16,df5b23c847d14e7a,35351b2c2ca8531d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:56:32.504 INFO 4486 --- [ocp-schedules-16,f634ba50690ced04,4ee74b131e808f61] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:56:32.506 INFO 4486 --- [ocp-schedules-16,f634ba50690ced04,4ee74b131e808f61] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:32.566 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.567 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:32.574 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.575 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:32.600 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.601 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:32.602 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:32.607 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.607 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:56:32.641 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:32.642 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.642 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:32.643 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:56:32.648 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.651 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:56:32.652 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:32.666 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.667 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:32.681 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.690 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:32.691 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.692 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:32.693 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:56:32.697 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:56:32.701 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:32.703 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:32.707 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:32.708 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.709 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:56:32.724 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:56:32.760 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:56:32.760 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:56:32.786 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:32.788 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.789 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:56:32.793 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.796 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:56:32.798 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:56:32.799 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:32.808 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:32.813 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.814 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:56:32.824 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:56:32.824 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:56:32.826 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.826 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:32.851 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.851 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:32.880 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.880 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:32.905 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.905 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:56:32.935 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.936 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:56:32.967 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:56:32.994 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:32.995 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:32.996 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:33.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:56:33.005 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:56:33.010 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:56:33.011 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:56:33.021 INFO 4486 --- [ocp-async-1,e722b1ad7642e090,efada35a32614f96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:56:33.037 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:56:33.180 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:56:33.207 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:33.208 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:33.209 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:56:33.241 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:56:33.241 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:56:33.348 INFO 4486 --- [ocp-async-3,c29b101eb34d7f29,c70a72372e252655] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:33.562 INFO 4486 --- [ocp-schedules-27,9c947d065aa0828d,88690ec0bd3d1d80] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:56:33.563 INFO 4486 --- [ocp-schedules-27,9c947d065aa0828d,88690ec0bd3d1d80] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:33.564 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:56:33.564 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:56:33.564 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:56:33.792 INFO 4486 --- [ocp-schedules-28,3a28984fa46c9a91,bda8776a677a6a7d] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=39968 2024-10-16 13:56:34.392 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:34.994 INFO 4486 --- [prometheus-minute-schedule-1,138217802d40dd22,c1a8cc4bfa1688ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:35.000 INFO 4486 --- [prometheus-minute-schedule-1,138217802d40dd22,c1a8cc4bfa1688ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:35.348 INFO 4486 --- [ocp-async-8,5e43d5351ef2ed67,f04d36b89007f405] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:35.875 INFO 4486 --- [ocp-schedules-2,ade120be21b752cb,fdddd41aee7380f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:56:35.913 WARN 4486 --- [scanner-13308150-1,ade120be21b752cb,68fe51b5faa0df96] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:56:37.063 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:56:37.064 INFO 4486 --- [ocp-schedules-25,f83a6aff1100407b,290b4720728f3483] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:56:37.066 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:56:37.066 INFO 4486 --- [ocp-schedules-24,9704425c444266d8,abc72a4668e3bdaa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:56:37.068 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:56:37.072 INFO 4486 --- [ocp-schedules-25,f83a6aff1100407b,290b4720728f3483] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:56:37.072 INFO 4486 --- [ocp-schedules-24,9704425c444266d8,abc72a4668e3bdaa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:37.081 INFO 4486 --- [ocp-schedules-11,40360c721ebfda8b,e0b6370d5eebc827] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:56:37.083 INFO 4486 --- [ocp-schedules-11,40360c721ebfda8b,e0b6370d5eebc827] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:37.084 INFO 4486 --- [ocp-schedules-21,d299710a91a93435,430f235cfebd4e1e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 13:56:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:56:37.086 INFO 4486 --- [ocp-schedules-21,d299710a91a93435,430f235cfebd4e1e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:56:37.090 INFO 4486 --- [ocp-schedules-2,64e3147622b182ad,93759d83a41a31b4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:56:37.091 INFO 4486 --- [ocp-schedules-5,ceb01c59f01626d1,ca375b9aba7e96c9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:56:37.091 INFO 4486 --- [host-check-clockdiff1,ceb01c59f01626d1,19b21a9bd1048fa6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:56:37.091 INFO 4486 --- [host-check-clockdiff2,ceb01c59f01626d1,ac09dc4e559b5938] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:56:37.092 INFO 4486 --- [ocp-schedules-10,64c486090a44973d,64e8b67791598f26] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:56:37.092 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:56:37.093 INFO 4486 --- [host-check-clockdiff3,ceb01c59f01626d1,04821b6b33fef8f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:56:37.093 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:56:37.094 INFO 4486 --- [host-check-clockdiff4,ceb01c59f01626d1,30383bb9c98f4e0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:56:37.094 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:56:37.095 INFO 4486 --- [host-check-agent-status1,64c486090a44973d,c56eaa883e76bbcb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:56:37.096 INFO 4486 --- [host-check-agent-status2,64c486090a44973d,cfd78fe994abd3c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:56:37.096 INFO 4486 --- [host-check-agent-status3,64c486090a44973d,6d927496c70093e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:56:37.097 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:56:37.097 INFO 4486 --- [host-check-agent-status4,64c486090a44973d,962fd873127fb52f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:56:37.101 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:37.101 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:56:37.102 INFO 4486 --- [host-check-agent-status1,64c486090a44973d,c56eaa883e76bbcb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:37.102 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:56:37.104 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:56:37.104 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:56:37.104 INFO 4486 --- [host-check-agent-status3,64c486090a44973d,6d927496c70093e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:56:37.106 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:37.106 INFO 4486 --- [host-check-agent-status1,64c486090a44973d,c56eaa883e76bbcb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:37.108 INFO 4486 --- [host-check-agent-status3,64c486090a44973d,6d927496c70093e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:56:37.108 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:56:37.129 INFO 4486 --- [host-check-clockdiff1,ceb01c59f01626d1,19b21a9bd1048fa6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058197 0 0 , error= 2024-10-16 13:56:37.130 INFO 4486 --- [host-check-agent-status2,64c486090a44973d,cfd78fe994abd3c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:56:37.130 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:56:37.153 INFO 4486 --- [host-check-clockdiff2,ceb01c59f01626d1,ac09dc4e559b5938] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058197 5 5 , error= 2024-10-16 13:56:37.154 INFO 4486 --- [host-check-agent-status2,64c486090a44973d,cfd78fe994abd3c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:56:37.154 INFO 4486 --- [host-check-agent-status4,64c486090a44973d,962fd873127fb52f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:56:37.156 INFO 4486 --- [host-check-agent-status4,64c486090a44973d,962fd873127fb52f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:56:37.157 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.158 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.158 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:56:37.158 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:37.159 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:56:37.173 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ac7de195a7f0368c,860ff2be570d1da5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=ac7de195a7f0368c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 13:56:37.174 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,ac7de195a7f0368c,860ff2be570d1da5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=ac7de195a7f0368c, duration=2 ms] 2024-10-16 13:56:37.176 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.177 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:56:37.180 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:56:37.182 INFO 4486 --- [host-check-agent-status1,64c486090a44973d,c56eaa883e76bbcb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:56:37.183 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:56:37.183 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:56:37.187629690+08:00 2024-10-16 13:56:37.183 INFO 4486 --- [host-check-agent-status3,64c486090a44973d,6d927496c70093e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:56:37.184 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:56:37.183+08:00 2024-10-16 13:56:37.184 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:56:37.185 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.185 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:56:37.185269600+08:00 2024-10-16 13:56:37.185 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:56:37.190147043+08:00 2024-10-16 13:56:37.186 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:56:37.186 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:56:37.185+08:00 2024-10-16 13:56:37.186 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:56:37.185+08:00 2024-10-16 13:56:37.186 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.186 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.187 INFO 4486 --- [host-check-agent-status2,64c486090a44973d,cfd78fe994abd3c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:56:37.187 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 13:56:37.188 INFO 4486 --- [host-check-agent-status1,64c486090a44973d,c56eaa883e76bbcb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:56:37.188 INFO 4486 --- [host-check-agent-status1,64c486090a44973d,c56eaa883e76bbcb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:56:37.188 INFO 4486 --- [host-check-agent-status1,64c486090a44973d,c56eaa883e76bbcb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:56:37.188 INFO 4486 --- [host-check-agent-status3,64c486090a44973d,6d927496c70093e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:56:37.188 INFO 4486 --- [host-check-agent-status3,64c486090a44973d,6d927496c70093e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:56:37.188 INFO 4486 --- [host-check-agent-status3,64c486090a44973d,6d927496c70093e8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:56:37.189 INFO 4486 --- [host-check-agent-status4,64c486090a44973d,962fd873127fb52f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:56:37.189 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:56:37.190 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 13:56:37.190 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:56:37.192 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:56:37.196273869+08:00 2024-10-16 13:56:37.192 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:56:37.192+08:00 2024-10-16 13:56:37.193 INFO 4486 --- [host-check-agent-status2,64c486090a44973d,cfd78fe994abd3c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:56:37.193 INFO 4486 --- [host-check-agent-status2,64c486090a44973d,cfd78fe994abd3c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:56:37.193 INFO 4486 --- [host-check-agent-status2,64c486090a44973d,cfd78fe994abd3c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:56:37.193 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:56:37.183+08:00 and status AVAILABLE 2024-10-16 13:56:37.195 INFO 4486 --- [host-check-agent-status4,64c486090a44973d,962fd873127fb52f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:56:37.195 INFO 4486 --- [host-check-agent-status4,64c486090a44973d,962fd873127fb52f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:56:37.195 INFO 4486 --- [host-check-agent-status4,64c486090a44973d,962fd873127fb52f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:56:37.195 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:56:37.192+08:00 and status AVAILABLE 2024-10-16 13:56:37.196 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:56:37.198 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:56:37.199 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:56:37.185+08:00 and status AVAILABLE 2024-10-16 13:56:37.200 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.200 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:56:37.185+08:00 and status AVAILABLE 2024-10-16 13:56:37.201 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:56:37.202 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:56:37.203 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.203 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.204 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:56:37.204 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 13:56:37.205 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:56:37.206 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:56:37.206 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.206 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.207 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:56:37.207 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:56:37.208 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:56:37.208 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:56:37.210 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:56:37.211 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:37.212 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:56:37.212 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:37.214 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.215 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:37.216 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.219 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.219 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.219 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.219 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.219 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.220 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 13:56:37.220 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.223 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.227 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 13:56:37.231 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.231 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:37.232 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.232 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:37.232 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:37.236 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:37.236 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 13:56:37.237 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:56:37.241 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:56:37.243 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.244 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:56:37.246 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.248 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:56:37.258 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.260 INFO 4486 --- [ocp-schedules-20,cd179a6b1f6439cf,4a68a1aeed37c3a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:56:37.265 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:56:37.268 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:56:37.270 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:56:37.277 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:56:37.280 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:56:37.281 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:56:37.290 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:56:37.302 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:56:37.305 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:37.306 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:37.308 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:37.317 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:56:37.335 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:56:37.345 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:56:37.348 INFO 4486 --- [ocp-async-4,83b60b0b73d005d4,a657021d4584422c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:37.349 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:56:37.354 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:56:37.361 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:56:37.362 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:37.366 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:56:37.377 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:37.380 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:56:37.383 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:37.390 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:56:37.392 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:56:37.392 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:37.403 INFO 4486 --- [host-check-host-status3,64e3147622b182ad,fa441a84022cd149] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:56:37.405 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:56:37.412 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:56:37.415 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:56:37.423 INFO 4486 --- [host-check-host-status4,64e3147622b182ad,429d3aa3b44f0540] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:56:37.426 INFO 4486 --- [host-check-host-status2,64e3147622b182ad,f3ceeaf4abc68602] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:56:37.452 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:37.454 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.455 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:56:37.455 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:56:37.457 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.457 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.459 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:56:37.464 INFO 4486 --- [host-check-host-status1,64e3147622b182ad,ffbf7a00038eac2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:56:37.468 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.468 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.468 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.468 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.468 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.469 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.496 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:56:37.542 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:56:37.582 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:37.596 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:37.597 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.598 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:56:37.600 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:56:37.606 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.607 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:56:37.608 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.608 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.609 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:56:37.618 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.618 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.618 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.618 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.618 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.619 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.628 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:56:37.846 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:56:37.892 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:37.894 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.895 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:56:37.897 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.897 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.898 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:56:37.908 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.908 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.908 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.908 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.908 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:37.908 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:37.926 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:56:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:56:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:56:37.994 INFO 4486 --- [prometheus-minute-schedule-1,c7d04872d5cc157c,cf61d94bdf33d818] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:37.997 INFO 4486 --- [ocp-schedules-3,deb8b7dff022fb81,1a8de0552bca4f2b] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 13:56:37.997 INFO 4486 --- [ocp-schedules-3,deb8b7dff022fb81,1a8de0552bca4f2b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 13:56:38.001 INFO 4486 --- [prometheus-minute-schedule-1,c7d04872d5cc157c,cf61d94bdf33d818] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:38.032 INFO 4486 --- [pool-overview-container-5,0cc299c0e0d2ef81,4ee270c5a679ab77] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:38.036 INFO 4486 --- [pool-10-thread-14,bbed0c20a4abe431,74ee1e081ac4ec0f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 13:56:38.040 INFO 4486 --- [pool-10-thread-15,bbed0c20a4abe431,a05ca43e10feb750] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 13:56:38.040 INFO 4486 --- [pool-10-thread-16,bbed0c20a4abe431,264aaf78bd566c40] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 13:56:38.041 INFO 4486 --- [pool-overview-container-5,0cc299c0e0d2ef81,4ee270c5a679ab77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.041 INFO 4486 --- [pool-10-thread-13,bbed0c20a4abe431,4560f650f7d8408a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 13:56:38.043 INFO 4486 --- [pool-overview-container-5,0cc299c0e0d2ef81,4ee270c5a679ab77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:56:38.047 INFO 4486 --- [ForkJoinPool-6-worker-14,bbed0c20a4abe431,7885a81b831c72ef] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 13:56:38.047 INFO 4486 --- [pool-10-thread-13,bbed0c20a4abe431,4560f650f7d8408a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 13:56:38.047 INFO 4486 --- [pool-10-thread-13,bbed0c20a4abe431,4560f650f7d8408a] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 13:56:38.047 INFO 4486 --- [ForkJoinPool-6-worker-14,bbed0c20a4abe431,7885a81b831c72ef] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 13:56:38.048 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 13:56:38.048 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 13:56:38.048 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 13:56:38.048 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 13:56:38.064 INFO 4486 --- [host-check-clockdiff3,ceb01c59f01626d1,04821b6b33fef8f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058198 5 5 , error= 2024-10-16 13:56:38.080 INFO 4486 --- [ForkJoinPool-6-worker-14,bbed0c20a4abe431,7885a81b831c72ef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.124 INFO 4486 --- [ForkJoinPool-6-worker-14,bbed0c20a4abe431,7885a81b831c72ef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:56:38.125 INFO 4486 --- [ForkJoinPool-6-worker-14,bbed0c20a4abe431,7885a81b831c72ef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.126 INFO 4486 --- [pool-10-thread-14,bbed0c20a4abe431,74ee1e081ac4ec0f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 13:56:38.126 INFO 4486 --- [ForkJoinPool-6-worker-14,bbed0c20a4abe431,7885a81b831c72ef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:56:38.136 INFO 4486 --- [pool-overview-container-5,0cc299c0e0d2ef81,4ee270c5a679ab77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.137 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.138 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:56:38.139 INFO 4486 --- [pool-overview-container-5,0cc299c0e0d2ef81,4ee270c5a679ab77] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 13:56:38.146 WARN 4486 --- [ForkJoinPool-6-worker-14,bbed0c20a4abe431,7885a81b831c72ef] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 13:56:38.153 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.153 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.154 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:56:38.154 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:56:38.158 INFO 4486 --- [pool-10-thread-14,bbed0c20a4abe431,74ee1e081ac4ec0f] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 13:56:38.159 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.160 INFO 4486 --- [ForkJoinPool-6-worker-9,bbed0c20a4abe431,d697bb8afd45c751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.160 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:56:38.160 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.162 INFO 4486 --- [ForkJoinPool-6-worker-9,bbed0c20a4abe431,d697bb8afd45c751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:56:38.163 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,1005c7f0cd359e7d,d0ae1e1e69b184ab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1005c7f0cd359e7d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:38.163 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,1005c7f0cd359e7d,d0ae1e1e69b184ab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1005c7f0cd359e7d, duration=1 ms] 2024-10-16 13:56:38.166 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:56:38.167 INFO 4486 --- [pool-10-thread-14,bbed0c20a4abe431,74ee1e081ac4ec0f] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 13:56:38.168 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.171 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:56:38.175 INFO 4486 --- [ForkJoinPool-6-worker-9,bbed0c20a4abe431,d697bb8afd45c751] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 13:56:38.176 INFO 4486 --- [ForkJoinPool-6-worker-9,bbed0c20a4abe431,d697bb8afd45c751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.178 INFO 4486 --- [ForkJoinPool-6-worker-9,bbed0c20a4abe431,d697bb8afd45c751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:56:38.179 INFO 4486 --- [ForkJoinPool-6-worker-2,bbed0c20a4abe431,8da9a08e91442ec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.179 INFO 4486 --- [ForkJoinPool-6-worker-9,bbed0c20a4abe431,d697bb8afd45c751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.181 INFO 4486 --- [ForkJoinPool-6-worker-9,bbed0c20a4abe431,d697bb8afd45c751] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:56:38.182 INFO 4486 --- [ForkJoinPool-6-worker-2,bbed0c20a4abe431,8da9a08e91442ec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:56:38.188 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.189 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:56:38.190 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:56:38.190 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.192 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:56:38.195 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.197 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:56:38.197 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.200 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:38.202 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:56:38.203 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.205 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:56:38.206 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.210 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.211 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:56:38.211 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:56:38.217 INFO 4486 --- [ForkJoinPool-6-worker-2,bbed0c20a4abe431,8da9a08e91442ec6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:56:38.218 INFO 4486 --- [ForkJoinPool-6-worker-2,bbed0c20a4abe431,8da9a08e91442ec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.220 INFO 4486 --- [ForkJoinPool-6-worker-2,bbed0c20a4abe431,8da9a08e91442ec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:56:38.226 INFO 4486 --- [ForkJoinPool-6-worker-2,bbed0c20a4abe431,8da9a08e91442ec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.227 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.227 INFO 4486 --- [ForkJoinPool-6-worker-2,bbed0c20a4abe431,8da9a08e91442ec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:56:38.230 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.232 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:56:38.233 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 13:56:38.236 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.236 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.239 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.240 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:56:38.240 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:56:38.243 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:56:38.244 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:56:38.246 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 13:56:38.254 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.255 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 13:56:38.258 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:56:38.260 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.261 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.261 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:38.261 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 13:56:38.263 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.264 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:56:38.264 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 13:56:38.266 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 13:56:38.267 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.267 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.267 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 13:56:38.274 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.274 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:56:38.276 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 13:56:38.277 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:56:38, 2024-10-16 13:56:38] 2024-10-16 13:56:38.298 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 13:56:38.300 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.301 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.302 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:56:38, 2024-10-16 13:56:38] 2024-10-16 13:56:38.304 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 13:56:38.304 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:38.305 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:38.306 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 13:56:38.306 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.307 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.307 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.308 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:56:38.309 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:56:38, 2024-10-16 13:56:38] 2024-10-16 13:56:38.313 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:56:38.316 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:38.316 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.317 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:56:38.319 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:56:38.320 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 13:56:38.320 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.321 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.323 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 13:56:38.323 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:56:38, 2024-10-16 13:56:38] 2024-10-16 13:56:38.324 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.330 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:38.330 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:38.330 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:38.330 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:38.330 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:56:38.330 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.330 INFO 4486 --- [ForkJoinPool-6-worker-0,bbed0c20a4abe431,f18916f356b9d072] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 13:56:38.331 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:38.331 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.333 INFO 4486 --- [pool-10-thread-15,bbed0c20a4abe431,a05ca43e10feb750] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 13:56:38.334 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 13:56:38.334 INFO 4486 --- [ForkJoinPool-6-worker-5,bbed0c20a4abe431,74c18ba27b57db26] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 13:56:38.337 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 13:56:38.342 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:56:38.344 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.344 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 13:56:38.346 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.348 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:56:38, 2024-10-16 13:56:38] 2024-10-16 13:56:38.354 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 13:56:38.372 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.378 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 13:56:38.378 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 13:56:38.380 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:56:38.381 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 13:56:38, 2024-10-16 13:56:38] 2024-10-16 13:56:38.384 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 13:56:38.384 INFO 4486 --- [ForkJoinPool-6-worker-7,bbed0c20a4abe431,068e1435a686cd20] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 13:56:38.387 INFO 4486 --- [pool-10-thread-16,bbed0c20a4abe431,264aaf78bd566c40] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 13:56:38.401 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:56:38.456 INFO 4486 --- [ocp-schedules-14,06befe97e54685f0,79b3de81a2b2604c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:56:38.456 INFO 4486 --- [ocp-schedules-14,06befe97e54685f0,79b3de81a2b2604c] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:56:38.457 INFO 4486 --- [ocp-schedules-14,06befe97e54685f0,79b3de81a2b2604c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:56:38.599 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:56:38.407944, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:56:38.599 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:56:38.407944, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:56:38.637 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:56:38.407944} } 2024-10-16 13:56:38.694 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:56:38.700 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:56:38.703 INFO 4486 --- [ocp-schedules-30,0033e37168f0f84e,9ada98275ba1e97f] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:56:38.901 INFO 4486 --- [host-check-clockdiff4,ceb01c59f01626d1,30383bb9c98f4e0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058198 5 5 , error= 2024-10-16 13:56:39.194 INFO 4486 --- [ocp-schedules-15,8a43975b3525ac28,6ec5b4a3c5f78461] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:56:39.348 INFO 4486 --- [ocp-async-7,2326f6c4debffb75,ee081d8603631625] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:40.006 INFO 4486 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:56:40.007 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:56:40.027 INFO 4486 --- [Check obproxy2,128ec0eff5b7b6ee,131c2f275ca4b7d1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:56:40.028 INFO 4486 --- [Check obproxy1,2d6c34f14106e4f0,836c229be64a0808] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:56:40.028 INFO 4486 --- [Collect obproxy1,faa540af1ab43d11,51a3e9bf8eb2e2b5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:56:40.028 INFO 4486 --- [Collect obproxy2,47d5b9eabfb2c70c,0e92d09b4b91324a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:56:40.035 INFO 4486 --- [Check obproxy2,128ec0eff5b7b6ee,131c2f275ca4b7d1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:40.036 INFO 4486 --- [Collect obproxy1,faa540af1ab43d11,51a3e9bf8eb2e2b5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:40.038 INFO 4486 --- [Collect obproxy2,47d5b9eabfb2c70c,0e92d09b4b91324a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:40.039 INFO 4486 --- [Check obproxy1,2d6c34f14106e4f0,836c229be64a0808] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:56:40.057 INFO 4486 --- [Collect obproxy2,47d5b9eabfb2c70c,0e92d09b4b91324a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:56:40.057 INFO 4486 --- [Check obproxy2,128ec0eff5b7b6ee,131c2f275ca4b7d1] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:56:40.057 INFO 4486 --- [Check obproxy2,128ec0eff5b7b6ee,131c2f275ca4b7d1] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:56:40.062 INFO 4486 --- [Collect obproxy2,47d5b9eabfb2c70c,0e92d09b4b91324a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:56:40.064 INFO 4486 --- [Check obproxy1,2d6c34f14106e4f0,836c229be64a0808] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:56:40.064 INFO 4486 --- [Check obproxy1,2d6c34f14106e4f0,836c229be64a0808] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:56:40.064 INFO 4486 --- [Collect obproxy1,faa540af1ab43d11,51a3e9bf8eb2e2b5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:56:40.066 INFO 4486 --- [Collect obproxy2,47d5b9eabfb2c70c,0e92d09b4b91324a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:56:40.069 INFO 4486 --- [Collect obproxy1,faa540af1ab43d11,51a3e9bf8eb2e2b5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:56:40.074 INFO 4486 --- [Collect obproxy1,faa540af1ab43d11,51a3e9bf8eb2e2b5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:56:40.232 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,c489e6ef1c567781,7585d8d277765eca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c489e6ef1c567781, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:40.232 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,c489e6ef1c567781,7585d8d277765eca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c489e6ef1c567781, duration=2 ms] 2024-10-16 13:56:40.393 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:40.994 INFO 4486 --- [prometheus-minute-schedule-1,f2c2e975a4c8ba3e,24fbbe66646a6ebf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:40.995 INFO 4486 --- [prometheus-hour-schedule-1,6c4590773fdb6df3,ca47a6ccf9264a3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:56:41.001 INFO 4486 --- [prometheus-minute-schedule-1,f2c2e975a4c8ba3e,24fbbe66646a6ebf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:41.112 INFO 4486 --- [ocp-schedules-30,dc9366f7258f7188,e0673d7ab832e8a5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:56:41.349 INFO 4486 --- [ocp-async-2,f97cd3ddb4f09fd8,9356e174d26b589f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:41.771 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,4a9575d5c891a265,96af8bbdf86dbd53] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4a9575d5c891a265, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:41.772 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,4a9575d5c891a265,96af8bbdf86dbd53] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4a9575d5c891a265, duration=2 ms] 2024-10-16 13:56:42.095 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,0ed214e085bf8a03,0438c37697f95411] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0ed214e085bf8a03, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:42.096 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,0ed214e085bf8a03,0438c37697f95411] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0ed214e085bf8a03, duration=2 ms] 2024-10-16 13:56:42.169 INFO 4486 --- [ocp-schedules-14,e7d3859a555bffa8,1382430fc2a46f09] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:56:43.021 INFO 4486 --- [ocp-async-6,3e391ea9af93840b,94dc4f758eb22750] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:56:43.349 INFO 4486 --- [ocp-async-4,6c27b507e01e73dd,c30936c0f94e5154] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:43.394 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:43.565 INFO 4486 --- [ocp-schedules-27,9c947d065aa0828d,88690ec0bd3d1d80] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:56:43.994 INFO 4486 --- [prometheus-minute-schedule-1,0ccb05785824659a,63c9033e502b690e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:44.000 INFO 4486 --- [prometheus-minute-schedule-1,0ccb05785824659a,63c9033e502b690e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:45.349 INFO 4486 --- [ocp-async-3,540ca4c5bcc1d1fb,65af6c87227abe46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:45.569 INFO 4486 --- [ocp-schedules-6,93fe89a46fdf8d0b,bc1058a4f5f2dfd1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:56:45.575 INFO 4486 --- [ocp-schedules-6,93fe89a46fdf8d0b,bc1058a4f5f2dfd1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:56:45.575 INFO 4486 --- [ocp-schedules-6,93fe89a46fdf8d0b,bc1058a4f5f2dfd1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:56:45.599 INFO 4486 --- [ocp-schedules-20,46e9a465a77cbd3f,fd21627948a3528a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:56:45.600 INFO 4486 --- [ocp-schedules-20,46e9a465a77cbd3f,fd21627948a3528a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:56:46.113 INFO 4486 --- [ocp-schedules-27,4fa16a2a2b7fa672,63d0ec32cead1124] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:56:46.149 WARN 4486 --- [scanner-13308471-1,4fa16a2a2b7fa672,cdd390db0dea77bd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:56:46.210 INFO 4486 --- [alarm-detect-5,4fa16a2a2b7fa672,911c06391f1cc04c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058175 ~ 1729058196 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058190, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058190, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058190=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:56:46.395 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:46.994 INFO 4486 --- [prometheus-minute-schedule-1,27b060eea60d8656,4756f131ff8ec2d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:47.001 INFO 4486 --- [prometheus-minute-schedule-1,27b060eea60d8656,4756f131ff8ec2d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:47.074 INFO 4486 --- [ocp-schedules-24,9704425c444266d8,abc72a4668e3bdaa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:56:47.191 INFO 4486 --- [ocp-schedules-11,40360c721ebfda8b,e0b6370d5eebc827] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:56:47.240 INFO 4486 --- [ocp-schedules-21,d299710a91a93435,430f235cfebd4e1e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 13:56:47.248 INFO 4486 --- [ocp-schedules-26,6ba50282b74f9ae5,8e6dd88a9765ce31] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:56:47.349 INFO 4486 --- [ocp-async-6,cef2e0f05754ff65,ceb34fed993f550f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:48.167 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,a1585e1f54fd2446,4e78f5f3b060db97] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a1585e1f54fd2446, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:48.167 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,a1585e1f54fd2446,4e78f5f3b060db97] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a1585e1f54fd2446, duration=1 ms] 2024-10-16 13:56:49.350 INFO 4486 --- [ocp-async-4,32e39bb4cb94befb,ec184c3f7567bd4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:49.396 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:49.994 INFO 4486 --- [prometheus-minute-schedule-1,bfbfdcdfb77124b1,b4f57134f3f2487f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:50.000 INFO 4486 --- [prometheus-minute-schedule-1,bfbfdcdfb77124b1,b4f57134f3f2487f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:50.238 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5b5552c66344399a,f224ecdeb8042065] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5b5552c66344399a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:50.238 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5b5552c66344399a,f224ecdeb8042065] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5b5552c66344399a, duration=1 ms] 2024-10-16 13:56:50.525 INFO 4486 --- [ocp-schedules-11,b36119f355a74034,5bf37cf6e88907b1] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:56:50.525 INFO 4486 --- [ocp-schedules-11,b36119f355a74034,5bf37cf6e88907b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:56:50.534 INFO 4486 --- [ocp-schedules-11,b36119f355a74034,5bf37cf6e88907b1] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:56:50.534 INFO 4486 --- [ocp-schedules-11,b36119f355a74034,5bf37cf6e88907b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:56:50.540 INFO 4486 --- [ocp-schedules-11,b36119f355a74034,5bf37cf6e88907b1] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:56:50.540 INFO 4486 --- [ocp-schedules-11,b36119f355a74034,5bf37cf6e88907b1] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703041150484287488 2024-10-16 13:56:50.543 INFO 4486 --- [ocp-schedules-11,b36119f355a74034,5bf37cf6e88907b1] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=56, maxSyncedRecordId=703043730111270912 2024-10-16 13:56:51.117 INFO 4486 --- [ocp-schedules-4,7a33d698b1abb120,8a83aa04c80dee81] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:56:51.216 INFO 4486 --- [ocp-schedules-25,e7ddbfade1a54ee2,60d3f30e0512d22f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:56:51.216 INFO 4486 --- [ocp-schedules-25,e7ddbfade1a54ee2,60d3f30e0512d22f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:56:51.216 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:56:51.216 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:56:51.217 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:56:51.217 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:56:51.216 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:56:51.221 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:56:51.221 INFO 4486 --- [ocp-schedules-25,e7ddbfade1a54ee2,60d3f30e0512d22f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:56:51.222 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:56:51.222 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:56:51.222 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:56:51.223 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:56:51.223 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:56:51.223 INFO 4486 --- [ocp-schedules-25,e7ddbfade1a54ee2,60d3f30e0512d22f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:56:51.223 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:56:51.223 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:56:51.223 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:56:51.225 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:56:51.246 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:56:51.246 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:56:51.246 INFO 4486 --- [ocp-schedules-25,e7ddbfade1a54ee2,60d3f30e0512d22f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:56:51.246 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:56:51.254 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:56:51.257 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:56:51.265 INFO 4486 --- [ocp-schedules-25,e7ddbfade1a54ee2,60d3f30e0512d22f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:56:51.350 INFO 4486 --- [ocp-async-3,69f5403878f1aaba,2e9467203c2d1b5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:51.776 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,33dfadb3b0423c94,00647ccc3f5b1543] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=33dfadb3b0423c94, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:51.776 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,33dfadb3b0423c94,00647ccc3f5b1543] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=33dfadb3b0423c94, duration=1 ms] 2024-10-16 13:56:52.102 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,c81170e79ab58500,e8fbecb109e83bc4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c81170e79ab58500, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:52.102 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,c81170e79ab58500,e8fbecb109e83bc4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c81170e79ab58500, duration=1 ms] 2024-10-16 13:56:52.396 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:52.994 INFO 4486 --- [prometheus-minute-schedule-1,5fffdcf2492bef77,e9678ea9cadbadb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:53.000 INFO 4486 --- [prometheus-minute-schedule-1,5fffdcf2492bef77,e9678ea9cadbadb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:53.022 INFO 4486 --- [ocp-async-7,d32ec216917e9891,e124f6ef3cf6af2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:56:53.350 INFO 4486 --- [ocp-async-6,46162f2321354ee1,2f46c4f48d7d6749] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:55.033 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a85a08c6f5fb72ed,3650cdb5afdb9048] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=a85a08c6f5fb72ed, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:56:55.034 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a85a08c6f5fb72ed,3650cdb5afdb9048] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=a85a08c6f5fb72ed, duration=3 ms] 2024-10-16 13:56:55.350 INFO 4486 --- [ocp-async-10,06492112ac427d4c,b31d40c1aafcb995] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:55.398 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:55.994 INFO 4486 --- [prometheus-minute-schedule-1,d8b82b61938b9ebb,1e6c8a6b3bb70671] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:56.001 INFO 4486 --- [prometheus-minute-schedule-1,d8b82b61938b9ebb,1e6c8a6b3bb70671] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:56.288 INFO 4486 --- [ocp-schedules-26,eb007a59da508279,b7122d4e2eeb6903] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:56:56.340 WARN 4486 --- [scanner-13308695-1,eb007a59da508279,6955691aba599e2d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:56:56.406 INFO 4486 --- [alarm-detect-4,eb007a59da508279,d59fb5cb65ab5dd7] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058190 ~ 1729058206 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058205, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058205, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058205=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:56:57.351 INFO 4486 --- [ocp-async-7,4b909c15c7d83a05,b1c0eeced7bb611a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:56:58.171 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f0940540ca76ce45,fdfd66f0bc285b0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f0940540ca76ce45, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:56:58.172 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f0940540ca76ce45,fdfd66f0bc285b0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f0940540ca76ce45, duration=2 ms] 2024-10-16 13:56:58.398 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:56:58.994 INFO 4486 --- [prometheus-minute-schedule-1,b008fcdb9428f3e3,ee5c984d13667312] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058220, interval=60 2024-10-16 13:56:58.999 INFO 4486 --- [prometheus-minute-schedule-1,b008fcdb9428f3e3,ee5c984d13667312] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:56:59.198 INFO 4486 --- [ocp-schedules-15,021f2a681e55c531,b5daf1a92f493e74] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:56:59.351 INFO 4486 --- [ocp-async-2,6dfb42b68a26074f,a33bf60715e62719] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:00.242 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d53c9e2028a04e30,c1010c6aa1e81710] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d53c9e2028a04e30, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:00.243 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,d53c9e2028a04e30,c1010c6aa1e81710] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d53c9e2028a04e30, duration=1 ms] 2024-10-16 13:57:01.123 INFO 4486 --- [ocp-schedules-27,3def82b968a8e545,e363a1552eb450c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:57:01.351 INFO 4486 --- [ocp-async-5,534c3569fc65c219,46cff8cd439d14ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:01.399 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:01.782 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,6cc5a5e87ddce735,8d4bd73a0b812926] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6cc5a5e87ddce735, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:01.782 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,6cc5a5e87ddce735,8d4bd73a0b812926] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6cc5a5e87ddce735, duration=1 ms] 2024-10-16 13:57:01.994 INFO 4486 --- [prometheus-minute-schedule-1,1dc720881ee9860c,d8bfe37516cfb215] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:01.999 INFO 4486 --- [prometheus-minute-schedule-1,1dc720881ee9860c,d8bfe37516cfb215] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:02.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,26b57436b52054a0,00bb374518b8f810] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=26b57436b52054a0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:57:02.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,36caf7d07b6dd074,027470d4ae1f5114] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=36caf7d07b6dd074, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:02.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,36caf7d07b6dd074,027470d4ae1f5114] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=36caf7d07b6dd074, duration=1 ms] 2024-10-16 13:57:02.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,26b57436b52054a0,00bb374518b8f810] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=26b57436b52054a0, duration=2 ms] 2024-10-16 13:57:02.429 INFO 4486 --- [ocp-schedules-12,e35052ec7f11b72b,a606d9d133e85077] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:57:02.429+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:57:02.433 INFO 4486 --- [ocp-schedules-12,e35052ec7f11b72b,a606d9d133e85077] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:57:02.439 INFO 4486 --- [ocp-schedules-12,e35052ec7f11b72b,a606d9d133e85077] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:57:02.454 INFO 4486 --- [ocp-schedules-12,e35052ec7f11b72b,a606d9d133e85077] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:57:03.022 INFO 4486 --- [ocp-async-6,da6619cd01e960ac,b804b6426eda9e9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:57:03.242 INFO 4486 --- [ocp-schedules-16,f634ba50690ced04,4ee74b131e808f61] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:57:03.242 INFO 4486 --- [ocp-schedules-16,ffac8afd0f8431df,cbbf418eefabe955] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:57:03.244 INFO 4486 --- [ocp-schedules-16,ffac8afd0f8431df,cbbf418eefabe955] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:03.274 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.274 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:03.298 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.307 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:03.310 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.311 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.311 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:57:03.330 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.338 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:03.341 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.342 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.343 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:03.344 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:57:03.344 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.347 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:57:03.349 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:03.351 INFO 4486 --- [ocp-async-9,0feb4a773ecb5f0b,838c80b7061429a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:03.405 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.405 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:03.417 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.427 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.428 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.429 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:03.430 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:57:03.436 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:57:03.441 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:03.443 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:03.468 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:57:03.508 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.510 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.510 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:57:03.534 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:57:03.535 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:57:03.543 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:57:03.554 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.555 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.555 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:57:03.565 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.566 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:57:03.566 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:57:03.567 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.568 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:03.571 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.574 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:57:03.577 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:03.611 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.612 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:03.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.643 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:03.668 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.669 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:03.693 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.693 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:03.717 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.717 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:03.741 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:03.766 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.767 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.768 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:03.773 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:57:03.777 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:57:03.782 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:57:03.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:57:03.806 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:57:03.951 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:57:03.979 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:03.981 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:03.982 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:57:04.015 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:57:04.015 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:57:04.400 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:04.994 INFO 4486 --- [prometheus-minute-schedule-1,6acec4e3fa9ae0af,dd26826f49e4f8a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:05.001 INFO 4486 --- [prometheus-minute-schedule-1,6acec4e3fa9ae0af,dd26826f49e4f8a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:05.351 INFO 4486 --- [ocp-async-8,5c9b5b26101c1620,208e6862902833bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:06.213 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dc8cfc5648b07a42,fb9ff658727f89ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=dc8cfc5648b07a42, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:57:06.214 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dc8cfc5648b07a42,fb9ff658727f89ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=dc8cfc5648b07a42, duration=3 ms] 2024-10-16 13:57:06.485 INFO 4486 --- [ocp-schedules-29,04f6a661286b0eb4,2f01864da8669aeb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:57:06.528 WARN 4486 --- [scanner-13309051-1,04f6a661286b0eb4,bc730b55a50cdf0e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:57:07.066 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:57:07.066 INFO 4486 --- [ocp-schedules-20,8805027ee54f3d7a,5d56303aeb9f21ea] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:57:07.068 INFO 4486 --- [ocp-schedules-20,8805027ee54f3d7a,5d56303aeb9f21ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:07.081 INFO 4486 --- [ocp-schedules-10,07a809654998ff53,e4992015bddda9d6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:57:07.083 INFO 4486 --- [ocp-schedules-10,07a809654998ff53,e4992015bddda9d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:57:07.089 INFO 4486 --- [ocp-schedules-12,17ef31d2aadc19f5,d6555940b81a1f41] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:57:07.089 INFO 4486 --- [ocp-schedules-9,661ff020558b4a74,ae9ae76c9fab3002] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:57:07.090 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:57:07.090 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:57:07.090 INFO 4486 --- [host-check-clockdiff1,661ff020558b4a74,ac821207cfa61cdb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:57:07.091 INFO 4486 --- [host-check-clockdiff2,661ff020558b4a74,2df1a369e7509042] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:57:07.091 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:57:07.091 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:57:07.091 INFO 4486 --- [host-check-clockdiff4,661ff020558b4a74,f596ec382abdec2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:57:07.091 INFO 4486 --- [host-check-clockdiff3,661ff020558b4a74,57760b006e1c8961] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:57:07.095 INFO 4486 --- [ocp-schedules-17,70615f2af09f49b3,d4566a9b04953697] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:57:07.096 INFO 4486 --- [host-check-agent-status1,70615f2af09f49b3,d1fc660d01eb02e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:57:07.096 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:07.097 INFO 4486 --- [host-check-agent-status2,70615f2af09f49b3,2f8297ecabd5dcfa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:57:07.097 INFO 4486 --- [host-check-agent-status3,70615f2af09f49b3,c346fb13b07422cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:57:07.097 INFO 4486 --- [host-check-agent-status4,70615f2af09f49b3,07608e960a93aa62] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:57:07.098 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:57:07.098 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:57:07.099 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:07.100 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:57:07.100 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:57:07.114 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:57:07.126 INFO 4486 --- [host-check-clockdiff1,661ff020558b4a74,ac821207cfa61cdb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058227 0 0 , error= 2024-10-16 13:57:07.126 INFO 4486 --- [host-check-agent-status1,70615f2af09f49b3,d1fc660d01eb02e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:07.126 INFO 4486 --- [host-check-clockdiff4,661ff020558b4a74,f596ec382abdec2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058227 4 4 , error= 2024-10-16 13:57:07.126 INFO 4486 --- [host-check-agent-status2,70615f2af09f49b3,2f8297ecabd5dcfa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:57:07.126 INFO 4486 --- [host-check-agent-status3,70615f2af09f49b3,c346fb13b07422cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:57:07.129 INFO 4486 --- [host-check-clockdiff3,661ff020558b4a74,57760b006e1c8961] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058227 5 5 , error= 2024-10-16 13:57:07.132 INFO 4486 --- [host-check-agent-status1,70615f2af09f49b3,d1fc660d01eb02e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:07.132 INFO 4486 --- [host-check-agent-status3,70615f2af09f49b3,c346fb13b07422cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:57:07.132 INFO 4486 --- [host-check-agent-status2,70615f2af09f49b3,2f8297ecabd5dcfa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:57:07.134 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:57:07.136 INFO 4486 --- [host-check-agent-status4,70615f2af09f49b3,07608e960a93aa62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:57:07.137 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.138 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:07.138 INFO 4486 --- [host-check-agent-status4,70615f2af09f49b3,07608e960a93aa62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:57:07.139 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:57:07.175 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:57:07.176 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:57:07.176 INFO 4486 --- [host-check-agent-status1,70615f2af09f49b3,d1fc660d01eb02e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:57:07.177 INFO 4486 --- [host-check-agent-status2,70615f2af09f49b3,2f8297ecabd5dcfa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:57:07.178 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:57:07.182424702+08:00 2024-10-16 13:57:07.179 INFO 4486 --- [host-check-agent-status3,70615f2af09f49b3,c346fb13b07422cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:57:07.179 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:57:07.178+08:00 2024-10-16 13:57:07.179 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:57:07.183626197+08:00 2024-10-16 13:57:07.180 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:57:07.179+08:00 2024-10-16 13:57:07.180 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:57:07.180 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status1,70615f2af09f49b3,d1fc660d01eb02e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status1,70615f2af09f49b3,d1fc660d01eb02e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status2,70615f2af09f49b3,2f8297ecabd5dcfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status1,70615f2af09f49b3,d1fc660d01eb02e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status3,70615f2af09f49b3,c346fb13b07422cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status2,70615f2af09f49b3,2f8297ecabd5dcfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status2,70615f2af09f49b3,2f8297ecabd5dcfa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status3,70615f2af09f49b3,c346fb13b07422cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-agent-status3,70615f2af09f49b3,c346fb13b07422cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:57:07.178+08:00 and status AVAILABLE 2024-10-16 13:57:07.184 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:57:07.185 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:07.186 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:57:07.179+08:00 and status AVAILABLE 2024-10-16 13:57:07.186 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:57:07.186024046+08:00 2024-10-16 13:57:07.186 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:57:07.187 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:57:07.186+08:00 2024-10-16 13:57:07.187 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:57:07.191322988+08:00 2024-10-16 13:57:07.188 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:57:07.187+08:00 2024-10-16 13:57:07.189 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:57:07.189 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:57:07.190 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:57:07.186+08:00 and status AVAILABLE 2024-10-16 13:57:07.191 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:57:07.187+08:00 and status AVAILABLE 2024-10-16 13:57:07.192 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.192 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:57:07.193 INFO 4486 --- [host-check-agent-status4,70615f2af09f49b3,07608e960a93aa62] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:57:07.194 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:57:07.195 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:57:07.196 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:57:07.197 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:57:07.198 INFO 4486 --- [host-check-agent-status4,70615f2af09f49b3,07608e960a93aa62] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:57:07.198 INFO 4486 --- [host-check-agent-status4,70615f2af09f49b3,07608e960a93aa62] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:57:07.198 INFO 4486 --- [host-check-agent-status4,70615f2af09f49b3,07608e960a93aa62] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:57:07.199 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:57:07.201 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:57:07.203 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:07.204 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:57:07.205 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.205 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:07.205 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:57:07.206 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:57:07.215 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.216 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:57:07.222 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:07.224 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:07.230 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.230 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:07.231 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:57:07.234 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.235 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:57:07.236 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:07.246 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:07.247 INFO 4486 --- [ocp-schedules-6,1f0526f9f6f8494f,4b7214f995d46332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:57:07.250 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:57:07.265 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:57:07.266 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:57:07.271 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:57:07.279 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:57:07.283 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:57:07.285 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:07.286 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:57:07.297 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:57:07.298 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:07.303 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:07.321 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:57:07.321 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:07.337 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:57:07.338 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:57:07.339 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:57:07.350 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:57:07.351 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:57:07.352 INFO 4486 --- [ocp-async-6,156b1cea815999a1,4be2376ea0dc9521] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:07.356 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:07.367 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:07.370 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:07.385 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:57:07.389 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:57:07.397 INFO 4486 --- [host-check-host-status3,17ef31d2aadc19f5,c70e0a608789e244] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:57:07.400 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:07.402 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:57:07.404 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:57:07.407 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:57:07.413 INFO 4486 --- [host-check-host-status4,17ef31d2aadc19f5,45538691d520329a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:57:07.421 INFO 4486 --- [host-check-host-status2,17ef31d2aadc19f5,ed715e42e4128d59] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:57:07.423 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:07.474 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:57:07.485 INFO 4486 --- [host-check-host-status1,17ef31d2aadc19f5,e58bf36234f1ef1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:57:07.994 INFO 4486 --- [prometheus-minute-schedule-1,896d995646a19fad,7a036103dd84ef0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:08.002 INFO 4486 --- [prometheus-minute-schedule-1,896d995646a19fad,7a036103dd84ef0f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:08.054 INFO 4486 --- [host-check-clockdiff2,661ff020558b4a74,2df1a369e7509042] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058228 5 5 , error= 2024-10-16 13:57:08.177 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,69b79a4e8cfca9ee,427639cb6a270eea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=69b79a4e8cfca9ee, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:08.177 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,69b79a4e8cfca9ee,427639cb6a270eea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=69b79a4e8cfca9ee, duration=1 ms] 2024-10-16 13:57:08.941 INFO 4486 --- [ocp-schedules-7,aab3ceaeea4f1cd1,283c353d5ec8a8eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:57:08.942 INFO 4486 --- [ocp-schedules-7,aab3ceaeea4f1cd1,283c353d5ec8a8eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:09.352 INFO 4486 --- [ocp-async-10,88567a3d145140b0,4a55b9405aa5d34e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:10.006 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:57:10.007 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:57:10.026 INFO 4486 --- [Check obproxy1,d0b0055754390ae2,278080467ffd6eaf] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:57:10.026 INFO 4486 --- [Check obproxy1,d0b0055754390ae2,278080467ffd6eaf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:10.028 INFO 4486 --- [Check obproxy2,dc796163a3c5424f,664d04f80dde2a04] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:57:10.028 INFO 4486 --- [Check obproxy2,dc796163a3c5424f,664d04f80dde2a04] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:10.032 INFO 4486 --- [Collect obproxy2,74056c061c167f73,d820bdaeaf2c5612] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:57:10.032 INFO 4486 --- [Collect obproxy2,74056c061c167f73,d820bdaeaf2c5612] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:10.034 INFO 4486 --- [Collect obproxy1,2fb23433f859f5d7,45fc092297ea9b6c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:57:10.034 INFO 4486 --- [Collect obproxy1,2fb23433f859f5d7,45fc092297ea9b6c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:10.053 INFO 4486 --- [Check obproxy2,dc796163a3c5424f,664d04f80dde2a04] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:57:10.053 INFO 4486 --- [Check obproxy2,dc796163a3c5424f,664d04f80dde2a04] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:57:10.054 INFO 4486 --- [Collect obproxy2,74056c061c167f73,d820bdaeaf2c5612] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:57:10.055 INFO 4486 --- [Check obproxy1,d0b0055754390ae2,278080467ffd6eaf] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:57:10.055 INFO 4486 --- [Check obproxy1,d0b0055754390ae2,278080467ffd6eaf] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:57:10.060 INFO 4486 --- [Collect obproxy2,74056c061c167f73,d820bdaeaf2c5612] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:57:10.062 INFO 4486 --- [Collect obproxy1,2fb23433f859f5d7,45fc092297ea9b6c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:57:10.098 INFO 4486 --- [Collect obproxy2,74056c061c167f73,d820bdaeaf2c5612] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:57:10.098 INFO 4486 --- [Collect obproxy1,2fb23433f859f5d7,45fc092297ea9b6c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:57:10.100 INFO 4486 --- [Collect obproxy1,2fb23433f859f5d7,45fc092297ea9b6c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:57:10.248 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,88b80fb74e9be81b,c537774cf11712e8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=88b80fb74e9be81b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:10.249 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,88b80fb74e9be81b,c537774cf11712e8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=88b80fb74e9be81b, duration=2 ms] 2024-10-16 13:57:10.401 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:10.980 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:57:10.994 INFO 4486 --- [prometheus-minute-schedule-1,c2ee3c884eed4119,18a07f2d409372c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:11.000 INFO 4486 --- [prometheus-minute-schedule-1,c2ee3c884eed4119,18a07f2d409372c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:57:11.127 INFO 4486 --- [ocp-schedules-28,07b0f940ff019e47,a55753547b3621f2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:57:11.352 INFO 4486 --- [ocp-async-9,698b0fd1aa5ae108,38a4efeabed05fed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:11.786 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,efecbcc48d20a120,8b43aeb0c87285e0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=efecbcc48d20a120, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:11.786 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,efecbcc48d20a120,8b43aeb0c87285e0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=efecbcc48d20a120, duration=1 ms] 2024-10-16 13:57:12.110 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c2bef7d5ed9d6f8c,3754aee8fd173c3b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c2bef7d5ed9d6f8c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:12.110 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c2bef7d5ed9d6f8c,3754aee8fd173c3b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c2bef7d5ed9d6f8c, duration=1 ms] 2024-10-16 13:57:13.022 INFO 4486 --- [ocp-async-1,8351b0beb5d56619,1a2f7471772db808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:57:13.352 INFO 4486 --- [ocp-async-4,81afcc4bdf9356d7,1e70b601b3c993b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:13.402 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:13.994 INFO 4486 --- [prometheus-minute-schedule-1,0b3b904e6147ac53,eb059c175095e8e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:14.001 INFO 4486 --- [prometheus-minute-schedule-1,0b3b904e6147ac53,eb059c175095e8e7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:14.365 ERROR 4486 --- [metric-archive-second-15,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '669741696473960448-1729058160' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:57:14.364269] [YB420AC531B8-00060B7D84EAB323-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '669741696473960448-1729058160' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:57:14.364269] [YB420AC531B8-00060B7D84EAB323-0-0] 2024-10-16 13:57:14.380 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058160, nodeCount=1000, affectRows=1000 2024-10-16 13:57:14.385 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058160, nodeCount=998, validCount=998 2024-10-16 13:57:14.404 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058160, nodeCount=1000, affectRows=1000 2024-10-16 13:57:14.407 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058160, nodeCount=1000, validCount=1000 2024-10-16 13:57:14.409 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058160, nodeCount=1000, affectRows=1000 2024-10-16 13:57:14.413 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058160, nodeCount=1000, validCount=1000 2024-10-16 13:57:14.622 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058160, nodeCount=506, affectRows=506 2024-10-16 13:57:14.623 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058160, nodeCount=505, validCount=505 2024-10-16 13:57:15.186 INFO 4486 --- [metric-archive-async-28,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058160, nodeCount=480, affectRows=480 2024-10-16 13:57:15.352 INFO 4486 --- [ocp-async-3,d16e9b616fde7ed9,bfbeefe70a6ace74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:15.791 INFO 4486 --- [metric-archive-async-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058160, nodeCount=480, validCount=480 2024-10-16 13:57:16.403 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:16.653 INFO 4486 --- [ocp-schedules-22,e741c5fbcd8e6168,0dd98f1a8dce90c0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:57:16.678 WARN 4486 --- [scanner-13310101-1,e741c5fbcd8e6168,d6966f224dc75bfe] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:57:16.733 INFO 4486 --- [alarm-detect-7,e741c5fbcd8e6168,7dac84f1f9cca266] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058205 ~ 1729058226 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058220, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058220, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058220=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:57:16.735 INFO 4486 --- [alarm-detect-7,e741c5fbcd8e6168,7dac84f1f9cca266] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 13:57:16.994 INFO 4486 --- [prometheus-minute-schedule-1,3347ef46b8a57986,0b151a8bdae46527] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:17.000 INFO 4486 --- [prometheus-minute-schedule-1,3347ef46b8a57986,0b151a8bdae46527] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:17.069 INFO 4486 --- [ocp-schedules-20,8805027ee54f3d7a,5d56303aeb9f21ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:57:17.222 INFO 4486 --- [ocp-schedules-10,07a809654998ff53,e4992015bddda9d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:57:17.251 INFO 4486 --- [ocp-schedules-25,1f924b0dc691b7e4,df1f7a99c234932b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:57:17.352 INFO 4486 --- [ocp-async-1,27609761250f7849,157468af0b446acc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:18.181 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,05fb9b174f520c71,269f940dfe5c167a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=05fb9b174f520c71, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:18.181 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,05fb9b174f520c71,269f940dfe5c167a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=05fb9b174f520c71, duration=1 ms] 2024-10-16 13:57:18.973 INFO 4486 --- [ocp-schedules-7,aab3ceaeea4f1cd1,283c353d5ec8a8eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:57:19.203 INFO 4486 --- [ocp-schedules-13,80f2fe58364137ba,1e866bed85e6e5de] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:57:19.353 INFO 4486 --- [ocp-async-8,f50a94b121086c67,7704d53a78c86146] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:19.404 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:19.994 INFO 4486 --- [prometheus-minute-schedule-1,d0d1392cfb0064e7,ae2fcc49e86297e7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:19.999 INFO 4486 --- [prometheus-minute-schedule-1,d0d1392cfb0064e7,ae2fcc49e86297e7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:20.253 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,830f4fde9e3b0275,8a04afc443ca4178] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=830f4fde9e3b0275, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:20.254 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,830f4fde9e3b0275,8a04afc443ca4178] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=830f4fde9e3b0275, duration=2 ms] 2024-10-16 13:57:20.869 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 13:57:20.870 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 13:57:20.996 INFO 4486 --- [prometheus-cache-monitor1,137a5a208b2f4728,16638185fbf409df] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13854,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8468} 2024-10-16 13:57:20.996 INFO 4486 --- [prometheus-cache-monitor1,137a5a208b2f4728,16638185fbf409df] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:57:21.133 INFO 4486 --- [ocp-schedules-21,d59e0a3bee15d3d5,cb575ca59cdfe841] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:57:21.353 INFO 4486 --- [ocp-async-6,9af31493f20577ba,6cb77a18c040483b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:21.791 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,4bd96566ff2e42c3,cd1f4763788cd269] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4bd96566ff2e42c3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:21.791 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,4bd96566ff2e42c3,cd1f4763788cd269] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4bd96566ff2e42c3, duration=1 ms] 2024-10-16 13:57:22.116 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dadd710e74a3a94d,b2ef8358e3ab2d3a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dadd710e74a3a94d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:22.116 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,dadd710e74a3a94d,b2ef8358e3ab2d3a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dadd710e74a3a94d, duration=2 ms] 2024-10-16 13:57:22.405 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:22.994 INFO 4486 --- [prometheus-minute-schedule-1,47d6e9d23812858e,5963deedd8e2ae37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:23.001 INFO 4486 --- [prometheus-minute-schedule-1,47d6e9d23812858e,5963deedd8e2ae37] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:23.022 INFO 4486 --- [ocp-async-4,4fd32517d7b7272e,f3fde7d352a07a7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:57:23.353 INFO 4486 --- [ocp-async-7,447bd11a770a211e,d80e6e7f171ff651] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:23.798 INFO 4486 --- [ocp-schedules-24,1e2ff879cb6b47e8,bfb0d4d694fd6e45] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:57:25.083 INFO 4486 --- [ocp-schedules-25,2a6d6a3e678f2135,c667618d294770de] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:57:25.353 INFO 4486 --- [ocp-async-5,ccb6352a0fc42f14,cf8aa4316740dd2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:25.406 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:25.994 INFO 4486 --- [prometheus-minute-schedule-1,55216e03656d66fc,4df9b17e4d91d959] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:25.999 INFO 4486 --- [prometheus-minute-schedule-1,55216e03656d66fc,4df9b17e4d91d959] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:26.459 INFO 4486 --- [ocp-schedules-30,6debdf98b4ef7e03,0b9a0ec71b39af19] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:57:26.459 INFO 4486 --- [ocp-schedules-30,6debdf98b4ef7e03,0b9a0ec71b39af19] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:57:26.500 INFO 4486 --- [Collect all tenants major compactions1,6debdf98b4ef7e03,f0619af281c3d1df] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:57:26.500 INFO 4486 --- [Collect all tenants major compactions5,6debdf98b4ef7e03,573643c16860b624] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:57:26.500 INFO 4486 --- [Collect all tenants major compactions3,6debdf98b4ef7e03,d9ced10804bce3d6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:57:26.501 INFO 4486 --- [Collect all tenants major compactions4,6debdf98b4ef7e03,14480d0c1d953419] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:57:26.501 INFO 4486 --- [Collect all tenants major compactions2,6debdf98b4ef7e03,1caddc6c118fff8d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:57:26.515 INFO 4486 --- [Collect all tenants major compactions2,6debdf98b4ef7e03,1caddc6c118fff8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:26.515 INFO 4486 --- [Collect all tenants major compactions1,6debdf98b4ef7e03,f0619af281c3d1df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:26.516 INFO 4486 --- [Collect all tenants major compactions2,6debdf98b4ef7e03,1caddc6c118fff8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:57:26.517 INFO 4486 --- [Collect all tenants major compactions1,6debdf98b4ef7e03,f0619af281c3d1df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:57:26.520 INFO 4486 --- [Collect all tenants major compactions3,6debdf98b4ef7e03,d9ced10804bce3d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:26.520 INFO 4486 --- [Collect all tenants major compactions1,6debdf98b4ef7e03,f0619af281c3d1df] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:57:26.520 INFO 4486 --- [Collect all tenants major compactions2,6debdf98b4ef7e03,1caddc6c118fff8d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:57:26.521 INFO 4486 --- [Collect all tenants major compactions2,6debdf98b4ef7e03,1caddc6c118fff8d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:57:26.521 INFO 4486 --- [Collect all tenants major compactions1,6debdf98b4ef7e03,f0619af281c3d1df] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:57:26.522 INFO 4486 --- [Collect all tenants major compactions3,6debdf98b4ef7e03,d9ced10804bce3d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:57:26.525 INFO 4486 --- [Collect all tenants major compactions3,6debdf98b4ef7e03,d9ced10804bce3d6] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:57:26.526 INFO 4486 --- [Collect all tenants major compactions3,6debdf98b4ef7e03,d9ced10804bce3d6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:57:26.532 INFO 4486 --- [Collect all tenants major compactions4,6debdf98b4ef7e03,14480d0c1d953419] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:26.533 INFO 4486 --- [Collect all tenants major compactions5,6debdf98b4ef7e03,573643c16860b624] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:26.534 INFO 4486 --- [Collect all tenants major compactions4,6debdf98b4ef7e03,14480d0c1d953419] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:57:26.534 INFO 4486 --- [Collect all tenants major compactions5,6debdf98b4ef7e03,573643c16860b624] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:57:26.537 INFO 4486 --- [Collect all tenants major compactions4,6debdf98b4ef7e03,14480d0c1d953419] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:57:26.537 INFO 4486 --- [Collect all tenants major compactions4,6debdf98b4ef7e03,14480d0c1d953419] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:57:26.539 INFO 4486 --- [Collect all tenants major compactions5,6debdf98b4ef7e03,573643c16860b624] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:57:26.539 INFO 4486 --- [Collect all tenants major compactions5,6debdf98b4ef7e03,573643c16860b624] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:57:26.540 INFO 4486 --- [ocp-schedules-30,6debdf98b4ef7e03,0b9a0ec71b39af19] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:57:26.887 INFO 4486 --- [ocp-schedules-24,27c6e5328c6cbc0a,64917522b6f973fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:57:26.915 WARN 4486 --- [scanner-13310420-1,27c6e5328c6cbc0a,52c04f6536fe6856] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:57:26.975 INFO 4486 --- [alarm-detect-3,27c6e5328c6cbc0a,2c448215a2c19209] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058220 ~ 1729058236 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058235, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058235, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058235=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:57:27.353 INFO 4486 --- [ocp-async-10,f3ea1ac0ed613bd8,8ab6672d03629975] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:27.727 INFO 4486 --- [ocp-schedules-10,0cfcd6e03a4783e1,b1acf97aa775e8e9] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:57:27.727+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:57:27.729 INFO 4486 --- [ocp-schedules-10,0cfcd6e03a4783e1,b1acf97aa775e8e9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:57:27.730 INFO 4486 --- [ocp-schedules-10,0cfcd6e03a4783e1,b1acf97aa775e8e9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:57:27.730 INFO 4486 --- [ocp-schedules-10,0cfcd6e03a4783e1,b1acf97aa775e8e9] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:57:28.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a144d75925a82981,937529006e4405d7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a144d75925a82981, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:28.186 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a144d75925a82981,937529006e4405d7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a144d75925a82981, duration=1 ms] 2024-10-16 13:57:28.372 INFO 4486 --- [ocp-schedules-1,209879f0d6f1ac0b,b9d5ec8b4b6c9754] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:57:28.407 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:28.994 INFO 4486 --- [prometheus-minute-schedule-1,3c5a9cc3cad9cde4,98432d4020ca5b40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:29.000 INFO 4486 --- [prometheus-minute-schedule-1,3c5a9cc3cad9cde4,98432d4020ca5b40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:29.354 INFO 4486 --- [ocp-async-3,5f610519abb9b101,9b0545d567331cef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:30.103 INFO 4486 --- [prometheus-second-schedule-1,dda865acba6551dc,5f5b5322f0452a74] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:57:30.261 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,501fcc96fc988729,03d7ee32f1bb4323] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=501fcc96fc988729, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:30.261 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,501fcc96fc988729,03d7ee32f1bb4323] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=501fcc96fc988729, duration=1 ms] 2024-10-16 13:57:31.138 INFO 4486 --- [ocp-schedules-18,e2527b58edeaf50b,b830b143b60fd4c2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:57:31.354 INFO 4486 --- [ocp-async-7,a2e92d23a1013e25,f74eade98bb29c61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:31.408 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:31.795 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,846418169afee49f,bd871e40bef2b003] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=846418169afee49f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:31.795 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,846418169afee49f,bd871e40bef2b003] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=846418169afee49f, duration=1 ms] 2024-10-16 13:57:31.994 INFO 4486 --- [prometheus-minute-schedule-1,1bda4a627ca5d8d4,e0582059d4370d94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:31.998 INFO 4486 --- [prometheus-minute-schedule-1,1bda4a627ca5d8d4,e0582059d4370d94] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:32.125 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,260c947dfff4aad7,1869990b05bc09b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=260c947dfff4aad7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:32.126 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,260c947dfff4aad7,1869990b05bc09b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=260c947dfff4aad7, duration=2 ms] 2024-10-16 13:57:33.022 INFO 4486 --- [ocp-async-2,0ed7832dbeead1ee,99cabbb682f3eb02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:57:33.354 INFO 4486 --- [ocp-async-4,52dcdb6a4c93c7f7,09d0d4aa27f5ec07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:34.015 INFO 4486 --- [ocp-schedules-16,ffac8afd0f8431df,cbbf418eefabe955] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:57:34.016 INFO 4486 --- [ocp-schedules-16,c906f781ea83b1d7,d51d0f39a996a70c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:57:34.017 INFO 4486 --- [ocp-schedules-16,c906f781ea83b1d7,d51d0f39a996a70c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:34.050 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.051 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:34.059 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.060 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:34.085 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.086 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:34.094 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.096 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.096 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:57:34.125 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.126 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.127 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:34.128 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:57:34.128 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.131 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:57:34.133 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:34.142 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.143 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:34.154 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.163 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.164 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.164 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:34.165 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:57:34.171 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:57:34.176 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:34.178 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:34.194 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.196 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.198 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:57:34.204 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:57:34.228 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:57:34.228 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:57:34.249 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.251 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.252 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:57:34.255 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.259 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:57:34.262 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:34.279 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:57:34.287 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.287 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:34.288 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.289 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.290 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:57:34.298 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:57:34.299 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:57:34.313 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.313 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:34.340 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.340 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:34.371 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.371 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:57:34.396 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.396 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:57:34.409 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:34.424 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:57:34.449 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.450 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.451 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:34.455 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:57:34.460 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:57:34.464 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:57:34.465 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:57:34.486 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:57:34.629 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:57:34.656 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:34.658 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:34.659 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:57:34.689 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:57:34.689 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:57:34.994 INFO 4486 --- [prometheus-minute-schedule-1,0200e7b77cd162da,831f579f6d26b1a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:35.001 INFO 4486 --- [prometheus-minute-schedule-1,0200e7b77cd162da,831f579f6d26b1a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:35.354 INFO 4486 --- [ocp-async-9,ddb0b57f56c9a848,3a1a6833d59eff4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:36.420 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,48cd7d4eb049a781,1dfaa32a5677c841] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=48cd7d4eb049a781, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:57:36.421 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,48cd7d4eb049a781,1dfaa32a5677c841] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=48cd7d4eb049a781, duration=2 ms] 2024-10-16 13:57:37.062 INFO 4486 --- [ocp-schedules-20,5ee7133961b1a26b,1b469dac24f144b1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:57:37.063 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:57:37.064 INFO 4486 --- [ocp-schedules-25,77cb7cd680e66ded,e08b44567f024f07] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:57:37.067 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:57:37.067 INFO 4486 --- [ocp-schedules-14,6e4f1d32db788684,94d5a62c8850da06] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:57:37.070 INFO 4486 --- [ocp-schedules-14,6e4f1d32db788684,94d5a62c8850da06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:37.071 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:57:37.072 INFO 4486 --- [ocp-schedules-25,77cb7cd680e66ded,e08b44567f024f07] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:57:37.082 INFO 4486 --- [ocp-schedules-3,37459b727e8f22b8,599a54d64e00bbb3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:57:37.083 INFO 4486 --- [ocp-schedules-3,37459b727e8f22b8,599a54d64e00bbb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:37.099 INFO 4486 --- [ocp-schedules-30,b6544018f89f8d73,2a5bf9b60f573864] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:57:37.099 INFO 4486 --- [ocp-schedules-19,2399bda659ce1913,3d5cd5805ffea967] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:57:37.099 INFO 4486 --- [ocp-schedules-10,2dd582033c4212ee,3502f60ebb02b91d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:57:37.115 WARN 4486 --- [scanner-13310660-1,5ee7133961b1a26b,06c23246ca6fd650] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:57:37.128 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:57:37.128 INFO 4486 --- [host-check-agent-status1,2399bda659ce1913,404b5d553fb1fefc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:57:37.128 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:57:37.129 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:57:37.130 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:57:37.131 INFO 4486 --- [host-check-clockdiff1,2dd582033c4212ee,56aa2b02ffa088d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:57:37.132 INFO 4486 --- [host-check-agent-status2,2399bda659ce1913,ebc74c0654f554b3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:57:37.132 INFO 4486 --- [host-check-agent-status3,2399bda659ce1913,06d95b301f9fb2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:57:37.140 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:37.140 INFO 4486 --- [host-check-agent-status3,2399bda659ce1913,06d95b301f9fb2a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:57:37.140 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:57:37.140 INFO 4486 --- [host-check-agent-status1,2399bda659ce1913,404b5d553fb1fefc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:37.140 INFO 4486 --- [host-check-agent-status2,2399bda659ce1913,ebc74c0654f554b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:57:37.141 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:57:37.141 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:57:37.144 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:57:37.144 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:57:37.144 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:37.144 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:57:37.145 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:57:37.145 INFO 4486 --- [host-check-agent-status2,2399bda659ce1913,ebc74c0654f554b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:57:37.146 INFO 4486 --- [host-check-agent-status3,2399bda659ce1913,06d95b301f9fb2a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:57:37.148 INFO 4486 --- [host-check-agent-status1,2399bda659ce1913,404b5d553fb1fefc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:37.162 INFO 4486 --- [host-check-agent-status4,2399bda659ce1913,36f7cf4d533b3de2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:57:37.166 INFO 4486 --- [host-check-clockdiff2,2dd582033c4212ee,27dc5addc375bdbf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:57:37.168 INFO 4486 --- [host-check-clockdiff3,2dd582033c4212ee,daec63ddf68099fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:57:37.198 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.202 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:37.203 INFO 4486 --- [host-check-agent-status4,2399bda659ce1913,36f7cf4d533b3de2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:57:37.208 INFO 4486 --- [host-check-clockdiff4,2dd582033c4212ee,468e37ea2f159a7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:57:37.220 INFO 4486 --- [host-check-clockdiff1,2dd582033c4212ee,56aa2b02ffa088d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058257 0 0 , error= 2024-10-16 13:57:37.210 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:57:37.222 INFO 4486 --- [host-check-agent-status4,2399bda659ce1913,36f7cf4d533b3de2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:57:37.302 INFO 4486 --- [host-check-clockdiff4,2dd582033c4212ee,468e37ea2f159a7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058257 4 4 , error= 2024-10-16 13:57:37.302 INFO 4486 --- [host-check-agent-status3,2399bda659ce1913,06d95b301f9fb2a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:57:37.302 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:57:37.302 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:57:37.305 INFO 4486 --- [host-check-agent-status1,2399bda659ce1913,404b5d553fb1fefc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:57:37.305 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:57:37.305 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:57:37.306 INFO 4486 --- [host-check-agent-status4,2399bda659ce1913,36f7cf4d533b3de2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:57:37.312 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:57:37.309847988+08:00 2024-10-16 13:57:37.312 INFO 4486 --- [host-check-agent-status3,2399bda659ce1913,06d95b301f9fb2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:57:37.312 INFO 4486 --- [host-check-agent-status3,2399bda659ce1913,06d95b301f9fb2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:57:37.312 INFO 4486 --- [host-check-agent-status3,2399bda659ce1913,06d95b301f9fb2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:57:37.312 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:57:37.312+08:00 2024-10-16 13:57:37.313 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:57:37.317242655+08:00 2024-10-16 13:57:37.314 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:57:37.313+08:00 2024-10-16 13:57:37.314 INFO 4486 --- [host-check-agent-status1,2399bda659ce1913,404b5d553fb1fefc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:57:37.312 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:57:37.309500562+08:00 2024-10-16 13:57:37.314 INFO 4486 --- [host-check-agent-status1,2399bda659ce1913,404b5d553fb1fefc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:57:37.314 INFO 4486 --- [host-check-agent-status1,2399bda659ce1913,404b5d553fb1fefc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:57:37.315 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:57:37.314+08:00 2024-10-16 13:57:37.316 INFO 4486 --- [host-check-agent-status2,2399bda659ce1913,ebc74c0654f554b3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:57:37.319 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.319 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.321 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:57:37.322 INFO 4486 --- [host-check-agent-status2,2399bda659ce1913,ebc74c0654f554b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:57:37.322 INFO 4486 --- [host-check-agent-status2,2399bda659ce1913,ebc74c0654f554b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:57:37.322 INFO 4486 --- [host-check-agent-status2,2399bda659ce1913,ebc74c0654f554b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:57:37.325 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:57:37.314+08:00 and status AVAILABLE 2024-10-16 13:57:37.325 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:57:37.312+08:00 and status AVAILABLE 2024-10-16 13:57:37.325 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:57:37.313+08:00 and status AVAILABLE 2024-10-16 13:57:37.325 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:57:37.328 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:57:37.328 INFO 4486 --- [host-check-agent-status4,2399bda659ce1913,36f7cf4d533b3de2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:57:37.328 INFO 4486 --- [host-check-agent-status4,2399bda659ce1913,36f7cf4d533b3de2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:57:37.328 INFO 4486 --- [host-check-agent-status4,2399bda659ce1913,36f7cf4d533b3de2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:57:37.333 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:57:37.333 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:57:37.333 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:57:37.334 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:57:37.334012938+08:00 2024-10-16 13:57:37.335 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:57:37.334+08:00 2024-10-16 13:57:37.335 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.336 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:57:37.338 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.339 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.339 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.339 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:57:37.334+08:00 and status AVAILABLE 2024-10-16 13:57:37.340 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:57:37.340 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:57:37.342 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:57:37.347 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:57:37.348 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:57:37.348 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:57:37.349 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:57:37.349 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:57:37.350 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:57:37.351 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.352 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:57:37.354 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.354 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.354 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.354 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.354 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.354 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.355 INFO 4486 --- [ocp-async-3,a00764c4e6d08695,42c6dda8ced13193] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:37.355 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:37.357 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:37.362 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.363 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:57:37.372 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.373 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:57:37.374 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:57:37.379 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:37.384 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:37.386 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:37.387 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.388 INFO 4486 --- [ocp-schedules-22,df1047d2a26e606d,c70b1eec9263bb4a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:57:37.388 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:57:37.409 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:57:37.409 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:37.416 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:57:37.421 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:57:37.424 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:57:37.430 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:57:37.434 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:57:37.449 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:37.451 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:37.456 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:37.457 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:57:37.469 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:57:37.478 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:57:37.483 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:57:37.489 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:57:37.490 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:57:37.498 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:57:37.500 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:57:37.506 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:37.508 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:57:37.516 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:37.523 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:37.534 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:57:37.536 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:57:37.544 INFO 4486 --- [host-check-host-status3,b6544018f89f8d73,9d36d97011f71174] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:57:37.546 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:57:37.546 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:57:37.556 INFO 4486 --- [host-check-host-status4,b6544018f89f8d73,e111e4f0e788ecbc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:57:37.561 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:57:37.566 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:57:37.576 INFO 4486 --- [host-check-host-status2,b6544018f89f8d73,09bbcff9b321142c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:57:37.610 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:37.612 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.614 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:57:37.615 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:57:37.615 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.615 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.616 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:57:37.625 INFO 4486 --- [host-check-host-status1,b6544018f89f8d73,ea8b3e81f853efc3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:57:37.634 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.634 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.634 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.634 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.634 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.634 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.669 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:57:37.708 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:57:37.752 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:37.768 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:37.769 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.770 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:57:37.771 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:57:37.772 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.773 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:57:37.774 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.775 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.776 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:57:37.784 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.784 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.784 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.784 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.784 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:37.785 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:37.792 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:57:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:57:37.987 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:57:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:57:37.994 INFO 4486 --- [prometheus-minute-schedule-1,8810cd09b02180bb,33798e80b82b9469] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:38.000 INFO 4486 --- [prometheus-minute-schedule-1,8810cd09b02180bb,33798e80b82b9469] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:38.038 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:38.042 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:38.043 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:57:38.044 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.045 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:38.045 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:57:38.054 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.054 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.054 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.054 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.054 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.054 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:38.073 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:57:38.130 INFO 4486 --- [host-check-clockdiff3,2dd582033c4212ee,daec63ddf68099fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058258 4 4 , error= 2024-10-16 13:57:38.165 INFO 4486 --- [host-check-clockdiff2,2dd582033c4212ee,27dc5addc375bdbf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058258 4 4 , error= 2024-10-16 13:57:38.191 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a416af71495352f9,10d717aee4d16f1c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a416af71495352f9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:38.191 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a416af71495352f9,10d717aee4d16f1c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a416af71495352f9, duration=1 ms] 2024-10-16 13:57:38.283 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:57:38.325 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:57:38.327 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:38.328 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:57:38.329 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.330 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:38.331 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:57:38.341 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.341 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.341 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.341 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.341 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:57:38.341 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:38.410 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:57:38.478 INFO 4486 --- [ocp-schedules-19,73e3e00ee13c9838,27e274aeab2fc5c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13854,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8468} 2024-10-16 13:57:38.478 INFO 4486 --- [ocp-schedules-19,73e3e00ee13c9838,27e274aeab2fc5c4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:57:38.606 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:57:38.41547, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:57:38.606 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:57:38.41547, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:57:38.646 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:57:38.41547} } 2024-10-16 13:57:38.700 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:57:38.705 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:57:38.708 INFO 4486 --- [ocp-schedules-5,156ddda32a048190,922940f12885e153] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:57:39.208 INFO 4486 --- [ocp-schedules-26,095571122d171b8d,53206276ff8b17e6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:57:39.355 INFO 4486 --- [ocp-async-7,72dbdc3395deb4ab,2c4d17cbe45f6302] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:40.007 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:57:40.007 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:57:40.026 INFO 4486 --- [Check obproxy1,781bf9ee554c7d02,e819e23d6364dc33] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:57:40.026 INFO 4486 --- [Check obproxy2,37abfd8438242d6a,db4b25342283919b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:57:40.026 INFO 4486 --- [Check obproxy1,781bf9ee554c7d02,e819e23d6364dc33] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:40.026 INFO 4486 --- [Check obproxy2,37abfd8438242d6a,db4b25342283919b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:40.031 INFO 4486 --- [Collect obproxy1,c6f620b3bfb4aa3b,5d65a38782429fe7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:57:40.031 INFO 4486 --- [Collect obproxy1,c6f620b3bfb4aa3b,5d65a38782429fe7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:40.031 INFO 4486 --- [Collect obproxy2,38b6448336f6380d,43a7af4bab8ee3d7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:57:40.031 INFO 4486 --- [Collect obproxy2,38b6448336f6380d,43a7af4bab8ee3d7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:57:40.050 INFO 4486 --- [Check obproxy2,37abfd8438242d6a,db4b25342283919b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:57:40.050 INFO 4486 --- [Check obproxy2,37abfd8438242d6a,db4b25342283919b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:57:40.056 INFO 4486 --- [Collect obproxy2,38b6448336f6380d,43a7af4bab8ee3d7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:57:40.060 INFO 4486 --- [Check obproxy1,781bf9ee554c7d02,e819e23d6364dc33] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:57:40.060 INFO 4486 --- [Check obproxy1,781bf9ee554c7d02,e819e23d6364dc33] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:57:40.061 INFO 4486 --- [Collect obproxy2,38b6448336f6380d,43a7af4bab8ee3d7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:57:40.063 INFO 4486 --- [Collect obproxy1,c6f620b3bfb4aa3b,5d65a38782429fe7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:57:40.066 INFO 4486 --- [Collect obproxy2,38b6448336f6380d,43a7af4bab8ee3d7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:57:40.069 INFO 4486 --- [Collect obproxy1,c6f620b3bfb4aa3b,5d65a38782429fe7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:57:40.074 INFO 4486 --- [Collect obproxy1,c6f620b3bfb4aa3b,5d65a38782429fe7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:57:40.265 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e86203967209f8f9,eb0dbcb94f20e28b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e86203967209f8f9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:40.265 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e86203967209f8f9,eb0dbcb94f20e28b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e86203967209f8f9, duration=1 ms] 2024-10-16 13:57:40.410 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:40.994 INFO 4486 --- [prometheus-minute-schedule-1,8f73ec4ec4951e5f,d1c0d82272d9cdd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:40.995 INFO 4486 --- [prometheus-hour-schedule-1,c1cfda1bf54e6e39,58a661a52e00efca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:57:41.001 INFO 4486 --- [prometheus-minute-schedule-1,8f73ec4ec4951e5f,d1c0d82272d9cdd6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:41.143 INFO 4486 --- [ocp-schedules-17,d870e792a2ffcb92,1a3e594b9de4bf65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:57:41.355 INFO 4486 --- [ocp-async-4,1c0935986e58e1cf,ac08a903d8888835] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:41.799 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2a60dc800d04479a,539579224cbea2b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2a60dc800d04479a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:41.799 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2a60dc800d04479a,539579224cbea2b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2a60dc800d04479a, duration=1 ms] 2024-10-16 13:57:42.130 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,084ca4a9bb95cbfe,ff5d055da25c568b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=084ca4a9bb95cbfe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:42.130 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,084ca4a9bb95cbfe,ff5d055da25c568b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=084ca4a9bb95cbfe, duration=1 ms] 2024-10-16 13:57:42.172 INFO 4486 --- [ocp-schedules-7,1ea704a327ca147d,55ab598d2b3aa7f2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:57:43.023 INFO 4486 --- [ocp-async-6,9f716207ca60f5af,6d9eaa903dd4ff9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:57:43.355 INFO 4486 --- [ocp-async-2,a873e081c05464be,2cf65672b50c6a7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:43.416 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:43.565 INFO 4486 --- [ocp-schedules-1,c8925d28289e3a25,7d493fe1c47ee89f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:57:43.567 INFO 4486 --- [ocp-schedules-1,c8925d28289e3a25,7d493fe1c47ee89f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:57:43.568 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:57:43.568 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:57:43.568 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:57:43.994 INFO 4486 --- [prometheus-minute-schedule-1,3da0c0b91e796beb,3aca8271f6949c93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:44.001 INFO 4486 --- [prometheus-minute-schedule-1,3da0c0b91e796beb,3aca8271f6949c93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:45.356 INFO 4486 --- [ocp-async-7,89d442ac4556530c,3829cec2d533639d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:45.575 INFO 4486 --- [ocp-schedules-7,57ce6b7bfcca8386,72badcf19f526b59] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:57:45.580 INFO 4486 --- [ocp-schedules-7,57ce6b7bfcca8386,72badcf19f526b59] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:57:45.580 INFO 4486 --- [ocp-schedules-7,57ce6b7bfcca8386,72badcf19f526b59] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:57:45.604 INFO 4486 --- [ocp-schedules-27,1d3f57d943f5f22e,affc7a09331df012] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:57:45.604 INFO 4486 --- [ocp-schedules-27,1d3f57d943f5f22e,affc7a09331df012] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:57:46.417 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:46.994 INFO 4486 --- [prometheus-minute-schedule-1,631e7c3b7201ec2b,25ec31cd309a823a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:47.000 INFO 4486 --- [prometheus-minute-schedule-1,631e7c3b7201ec2b,25ec31cd309a823a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:47.104 INFO 4486 --- [ocp-schedules-14,6e4f1d32db788684,94d5a62c8850da06] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:57:47.255 INFO 4486 --- [ocp-schedules-12,9085c047d4762ea1,37b330b1cb3933b1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:57:47.329 INFO 4486 --- [ocp-schedules-3,37459b727e8f22b8,599a54d64e00bbb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:57:47.356 INFO 4486 --- [ocp-async-4,e0d52b3f07e3ed30,9abb7c5bc3394971] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:47.455 INFO 4486 --- [ocp-schedules-21,fede834294ced9fb,00cd317f38617896] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:57:47.472 WARN 4486 --- [scanner-13310947-1,fede834294ced9fb,46750ad38e23a9a4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:57:47.555 INFO 4486 --- [alarm-detect-15,fede834294ced9fb,b10eeef94e6d6ed5] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058235 ~ 1729058257 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058250, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058250, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058250=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:57:48.197 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1917873088267b05,1c1be76d5692fc33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1917873088267b05, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:48.198 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1917873088267b05,1c1be76d5692fc33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1917873088267b05, duration=2 ms] 2024-10-16 13:57:48.759 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:48.760 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:57:48.761 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:57:48.762 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:57:48.763 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:57:49.356 INFO 4486 --- [ocp-async-9,eec3f01feed4f909,a03ba82e8df392e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:49.418 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:49.980 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,56a05fbc8b33b731,f9bf8acf2cf9ca80] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=56a05fbc8b33b731, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:57:49.982 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,56a05fbc8b33b731,f9bf8acf2cf9ca80] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=56a05fbc8b33b731, duration=4 ms] 2024-10-16 13:57:49.994 INFO 4486 --- [prometheus-minute-schedule-1,92eaf8e93d819be3,a7b94e9e3ca15ada] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:50.001 INFO 4486 --- [prometheus-minute-schedule-1,92eaf8e93d819be3,a7b94e9e3ca15ada] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:50.272 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a3cea8977a7e4275,887faec8e5f9b3c9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a3cea8977a7e4275, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:50.272 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,a3cea8977a7e4275,887faec8e5f9b3c9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a3cea8977a7e4275, duration=1 ms] 2024-10-16 13:57:50.544 INFO 4486 --- [ocp-schedules-22,641161893077be61,f4374e435f5b8a71] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:57:50.544 INFO 4486 --- [ocp-schedules-22,641161893077be61,f4374e435f5b8a71] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:57:50.546 INFO 4486 --- [ocp-schedules-22,641161893077be61,f4374e435f5b8a71] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:57:50.546 INFO 4486 --- [ocp-schedules-22,641161893077be61,f4374e435f5b8a71] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:57:50.546 INFO 4486 --- [ocp-schedules-22,641161893077be61,f4374e435f5b8a71] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:57:50.546 INFO 4486 --- [ocp-schedules-22,641161893077be61,f4374e435f5b8a71] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 13:57:50.824 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 13:57:50.824 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 13:57:50.839 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 13:57:50.839 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 13:57:50.839 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 13:57:50.853 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 13:57:50.853 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 13:57:50.853 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 13:57:50.860 INFO 4486 --- [ocp-schedules-14,6247143413917282,19396bbe56fab024] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 13:57:51.149 INFO 4486 --- [ocp-schedules-13,540139d8ac5cdd38,e59d792e7efbfe2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:57:51.272 INFO 4486 --- [ocp-schedules-29,ced0ab95e3269846,ba34e8a7dcc16cba] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:57:51.272 INFO 4486 --- [ocp-schedules-29,ced0ab95e3269846,ba34e8a7dcc16cba] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:57:51.272 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:57:51.272 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:57:51.272 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:57:51.272 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:57:51.272 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:57:51.278 INFO 4486 --- [ocp-schedules-29,ced0ab95e3269846,ba34e8a7dcc16cba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:57:51.278 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:57:51.278 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:57:51.279 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:57:51.280 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:57:51.281 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:57:51.281 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:57:51.281 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:57:51.281 INFO 4486 --- [ocp-schedules-29,ced0ab95e3269846,ba34e8a7dcc16cba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:57:51.282 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:57:51.282 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:57:51.284 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:57:51.307 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:57:51.307 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:57:51.308 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:57:51.309 INFO 4486 --- [ocp-schedules-29,ced0ab95e3269846,ba34e8a7dcc16cba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:57:51.311 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:57:51.325 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:57:51.328 INFO 4486 --- [ocp-schedules-29,ced0ab95e3269846,ba34e8a7dcc16cba] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:57:51.356 INFO 4486 --- [ocp-async-5,8b1db1f819f82b25,13112235116e47a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:51.803 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3028a95ef5526180,a2551fceff757655] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3028a95ef5526180, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:51.803 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3028a95ef5526180,a2551fceff757655] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3028a95ef5526180, duration=1 ms] 2024-10-16 13:57:52.134 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4a75d54c07692612,3c73747b306ef41c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4a75d54c07692612, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:52.134 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4a75d54c07692612,3c73747b306ef41c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4a75d54c07692612, duration=1 ms] 2024-10-16 13:57:52.419 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:52.994 INFO 4486 --- [prometheus-minute-schedule-1,3ac322bba7e95b60,af9a1d9568d7bfff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:53.000 INFO 4486 --- [prometheus-minute-schedule-1,3ac322bba7e95b60,af9a1d9568d7bfff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:53.023 INFO 4486 --- [ocp-async-2,fbdf16ed123d1850,1572b25e8612807e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:57:53.356 INFO 4486 --- [ocp-async-6,2293fc10d801a1a3,bd36d9a0ba0e4b91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:53.568 INFO 4486 --- [ocp-schedules-1,c8925d28289e3a25,7d493fe1c47ee89f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:57:55.357 INFO 4486 --- [ocp-async-3,6f7ab048cc61c1a4,9764a06c8341f0d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:55.420 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:55.994 INFO 4486 --- [prometheus-minute-schedule-1,533f0b98a8ba9c96,8b081a3dd3f01188] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:56.000 INFO 4486 --- [prometheus-minute-schedule-1,533f0b98a8ba9c96,8b081a3dd3f01188] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:57.357 INFO 4486 --- [ocp-async-2,c10d8e4122a7c6b4,66d96edcefe25c15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:57:57.633 INFO 4486 --- [ocp-schedules-13,0d2ef556a5db7051,4abf6351d6f65840] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:57:57.696 WARN 4486 --- [scanner-13311243-1,0d2ef556a5db7051,b8914084f0342b1b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:57:57.760 INFO 4486 --- [alarm-detect-12,0d2ef556a5db7051,e6bc3d69a8ec4b1c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058250 ~ 1729058267 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058265, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058265, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058265=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:57:58.207 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e2e75b64a11cf327,e6d2b2adfebf6b1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e2e75b64a11cf327, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:57:58.207 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e2e75b64a11cf327,e6d2b2adfebf6b1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e2e75b64a11cf327, duration=1 ms] 2024-10-16 13:57:58.420 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:57:58.994 INFO 4486 --- [prometheus-minute-schedule-1,f86d5395905f8f78,69342e3cec66a9cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058280, interval=60 2024-10-16 13:57:59.000 INFO 4486 --- [prometheus-minute-schedule-1,f86d5395905f8f78,69342e3cec66a9cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:57:59.212 INFO 4486 --- [ocp-schedules-7,321bc36a1d309fb2,ec098c9af7cd4974] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:57:59.274 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f02bc26c425f94ef,6a46de14ac29ca14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=f02bc26c425f94ef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:57:59.274 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,f02bc26c425f94ef,6a46de14ac29ca14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=f02bc26c425f94ef, duration=1 ms] 2024-10-16 13:57:59.357 INFO 4486 --- [ocp-async-10,6b31f9ef84e7d1a0,96314226c999143d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:00.277 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9165ebb0c6245952,c50561dc18c8032e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9165ebb0c6245952, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:00.277 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9165ebb0c6245952,c50561dc18c8032e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9165ebb0c6245952, duration=1 ms] 2024-10-16 13:58:00.359 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 13:58:00.376 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:00.377 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:00.404 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:00.405 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:00.430 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 13:58:00.431 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 13:58:00.456 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 13:58:00.457 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 13:58:00.479 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:00.480 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:00.502 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:58:00.503 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:58:00.529 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:58:00.531 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:58:00.553 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 13:58:00.555 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 13:58:00.577 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:58:00.579 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:58:00.603 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 13:58:00.606 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 13:58:00.630 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:00.631 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:00.654 INFO 4486 --- [ocp-schedules-17,ee6f946899052549,d8001a528fa33e68] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 13:58:01.155 INFO 4486 --- [ocp-schedules-2,63c44191f9a882ad,959c5bf25b5b5ad1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:58:01.357 INFO 4486 --- [ocp-async-1,97bf43670637c25d,d2ca62252e995031] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:01.421 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:01.807 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b81c65ba0fae5bce,0381cd183f7c679f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b81c65ba0fae5bce, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:01.807 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,b81c65ba0fae5bce,0381cd183f7c679f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b81c65ba0fae5bce, duration=1 ms] 2024-10-16 13:58:01.994 INFO 4486 --- [prometheus-minute-schedule-1,3af545f0c7d23e54,a9a2894f8af779a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:02.001 INFO 4486 --- [prometheus-minute-schedule-1,3af545f0c7d23e54,a9a2894f8af779a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:02.139 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,256545569c990272,713e113d29634569] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=256545569c990272, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:02.139 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,256545569c990272,713e113d29634569] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=256545569c990272, duration=1 ms] 2024-10-16 13:58:02.485 INFO 4486 --- [ocp-schedules-11,c176dde2ea78e5d2,0f634a9c769b6069] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:58:02.485+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:58:02.489 INFO 4486 --- [ocp-schedules-11,c176dde2ea78e5d2,0f634a9c769b6069] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:58:02.493 INFO 4486 --- [ocp-schedules-11,c176dde2ea78e5d2,0f634a9c769b6069] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:58:02.499 INFO 4486 --- [ocp-schedules-11,c176dde2ea78e5d2,0f634a9c769b6069] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster glob_3zone for slowSqlStat, sql count is 0. 2024-10-16 13:58:03.023 INFO 4486 --- [ocp-async-6,a22591391b8698a4,33ec51f26dae0c67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:58:03.357 INFO 4486 --- [ocp-async-7,478b5ce13ad5f6b3,60da6e784f01a844] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:04.422 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:04.690 INFO 4486 --- [ocp-schedules-16,c906f781ea83b1d7,d51d0f39a996a70c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:58:04.690 INFO 4486 --- [ocp-schedules-16,c6b837e24a8a2466,e78e926cb32f336d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:58:04.693 INFO 4486 --- [ocp-schedules-16,c6b837e24a8a2466,e78e926cb32f336d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:04.720 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.721 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:04.737 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.738 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:04.755 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:04.755 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.756 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:04.756 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.757 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:58:04.785 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:04.786 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.786 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:04.787 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:58:04.788 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.791 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:58:04.792 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:04.801 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:04.801 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:04.811 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:04.821 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:04.822 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.823 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:04.824 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:58:04.829 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:58:04.835 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:04.836 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:04.858 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:04.859 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.860 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:58:04.861 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:58:04.915 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:58:04.915 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:58:04.915 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:04.917 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.918 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:04.922 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.925 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:58:04.928 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:04.951 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:04.951 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:04.957 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:58:04.969 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:04.970 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:04.970 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:58:04.979 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:04.979 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:04.981 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:58:04.981 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057686917000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:58:04.994 INFO 4486 --- [prometheus-minute-schedule-1,261551fff1583b5e,86e5b1b72461bad5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:05.006 INFO 4486 --- [prometheus-minute-schedule-1,261551fff1583b5e,86e5b1b72461bad5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:05.006 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:05.006 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:05.041 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:05.041 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:05.066 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:05.066 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:05.090 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:05.117 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:05.119 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:05.120 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:05.124 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:58:05.129 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:58:05.133 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:58:05.134 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:58:05.156 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:58:05.296 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:58:05.323 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:05.325 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:05.326 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:58:05.356 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:58:05.356 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:58:05.358 INFO 4486 --- [ocp-async-4,4186e53e17e38c81,1430a0659eb044ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:06.917 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,68ef250897b37ece,3deeb74f2ff61bf0] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&version=2&ObClusterId=1701313637, client=10.197.49.184, traceId=68ef250897b37ece, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=OceanBase,1701313637,2,{"ObRegion":"OceanBase","ObCluster":"OceanBase","ObRegionId":1701313637,"ObClusterId":1701313637,"Type":"PRIMARY","timestamp":1729058286911636,"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:58:06.921 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,68ef250897b37ece,3deeb74f2ff61bf0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=OceanBase, obClusterId=1701313637, rootServerJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, groupName=OceanBase, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:58:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,68ef250897b37ece,3deeb74f2ff61bf0] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:58:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,68ef250897b37ece,3deeb74f2ff61bf0] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=OceanBase, clusterId=1701313637, affectRows=1 2024-10-16 13:58:06.922 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,68ef250897b37ece,3deeb74f2ff61bf0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.184, traceId=68ef250897b37ece, duration=9 ms] 2024-10-16 13:58:07.066 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:58:07.066 INFO 4486 --- [ocp-schedules-29,48bf592a2a1b27ca,c16030ca6ccaf9b3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:58:07.068 INFO 4486 --- [ocp-schedules-29,48bf592a2a1b27ca,c16030ca6ccaf9b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:07.081 INFO 4486 --- [ocp-schedules-22,4e81fd7615995859,fd81417f0dce7a6b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:58:07.083 INFO 4486 --- [ocp-schedules-22,4e81fd7615995859,fd81417f0dce7a6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:58:07.089 INFO 4486 --- [ocp-schedules-7,58249f092df598ed,64c17ce07831aa6d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:58:07.089 INFO 4486 --- [ocp-schedules-13,92e70bf3ae64ac0c,eb6cae0f0e11a4f2] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:58:07.089 INFO 4486 --- [ocp-schedules-27,871b83910e04a8a9,076506abb143a460] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:58:07.090 INFO 4486 --- [host-check-clockdiff1,58249f092df598ed,a21ab77d36688928] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:58:07.090 INFO 4486 --- [host-check-clockdiff2,58249f092df598ed,cb605dc7d8350db0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:58:07.090 INFO 4486 --- [host-check-clockdiff3,58249f092df598ed,d5597a6e2a5f292a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:58:07.090 INFO 4486 --- [host-check-agent-status1,871b83910e04a8a9,0048aa8ba5fc2ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:58:07.091 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:58:07.091 INFO 4486 --- [host-check-clockdiff4,58249f092df598ed,b9c2321e1896f1e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:58:07.092 INFO 4486 --- [host-check-agent-status2,871b83910e04a8a9,d54a921a4bf9f8d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:58:07.092 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:58:07.094 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:58:07.094 INFO 4486 --- [host-check-agent-status3,871b83910e04a8a9,0375c8052479cf80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:58:07.095 INFO 4486 --- [host-check-agent-status4,871b83910e04a8a9,f6bfb84d2eea17c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:58:07.095 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:58:07.097 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:07.097 INFO 4486 --- [host-check-agent-status1,871b83910e04a8a9,0048aa8ba5fc2ad3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:07.127 INFO 4486 --- [host-check-agent-status1,871b83910e04a8a9,0048aa8ba5fc2ad3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:07.127 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:07.127 INFO 4486 --- [host-check-clockdiff2,58249f092df598ed,cb605dc7d8350db0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058287 4 4 , error= 2024-10-16 13:58:07.133 INFO 4486 --- [host-check-clockdiff1,58249f092df598ed,a21ab77d36688928] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058287 0 0 , error= 2024-10-16 13:58:07.133 INFO 4486 --- [host-check-clockdiff3,58249f092df598ed,d5597a6e2a5f292a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058287 5 5 , error= 2024-10-16 13:58:07.139 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:07.139 INFO 4486 --- [host-check-agent-status2,871b83910e04a8a9,d54a921a4bf9f8d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:07.139 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:07.140 INFO 4486 --- [host-check-clockdiff4,58249f092df598ed,b9c2321e1896f1e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058287 4 4 , error= 2024-10-16 13:58:07.140 INFO 4486 --- [host-check-agent-status4,871b83910e04a8a9,f6bfb84d2eea17c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:07.142 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:07.142 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:07.142 INFO 4486 --- [host-check-agent-status3,871b83910e04a8a9,0375c8052479cf80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:07.143 INFO 4486 --- [host-check-agent-status2,871b83910e04a8a9,d54a921a4bf9f8d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:07.144 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:07.144 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:07.144 INFO 4486 --- [host-check-agent-status3,871b83910e04a8a9,0375c8052479cf80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:07.145 INFO 4486 --- [host-check-agent-status4,871b83910e04a8a9,f6bfb84d2eea17c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:07.146 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.147 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:07.148 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:58:07.166 INFO 4486 --- [host-check-agent-status2,871b83910e04a8a9,d54a921a4bf9f8d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:58:07.167 INFO 4486 --- [host-check-agent-status1,871b83910e04a8a9,0048aa8ba5fc2ad3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:58:07.167 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:58:07.171 INFO 4486 --- [host-check-agent-status3,871b83910e04a8a9,0375c8052479cf80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:58:07.174 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:58:07.174 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:58:07.173724351+08:00 2024-10-16 13:58:07.175 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:58:07.175 INFO 4486 --- [host-check-agent-status4,871b83910e04a8a9,f6bfb84d2eea17c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:58:07.175 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:58:07.174+08:00 2024-10-16 13:58:07.176 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:58:07.176 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:58:07.181109965+08:00 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status2,871b83910e04a8a9,d54a921a4bf9f8d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status2,871b83910e04a8a9,d54a921a4bf9f8d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status2,871b83910e04a8a9,d54a921a4bf9f8d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status1,871b83910e04a8a9,0048aa8ba5fc2ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status1,871b83910e04a8a9,0048aa8ba5fc2ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status1,871b83910e04a8a9,0048aa8ba5fc2ad3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status3,871b83910e04a8a9,0375c8052479cf80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status3,871b83910e04a8a9,0375c8052479cf80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-agent-status3,871b83910e04a8a9,0375c8052479cf80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:58:07.177 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:58:07.177+08:00 2024-10-16 13:58:07.177 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:58:07.181399311+08:00 2024-10-16 13:58:07.178 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:58:07.182657075+08:00 2024-10-16 13:58:07.178 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:58:07.177+08:00 2024-10-16 13:58:07.179 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:58:07.178+08:00 2024-10-16 13:58:07.180 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:58:07.174+08:00 and status AVAILABLE 2024-10-16 13:58:07.180 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:58:07.177+08:00 and status AVAILABLE 2024-10-16 13:58:07.181 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:58:07.178+08:00 and status AVAILABLE 2024-10-16 13:58:07.181 INFO 4486 --- [host-check-agent-status4,871b83910e04a8a9,f6bfb84d2eea17c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:58:07.181 INFO 4486 --- [host-check-agent-status4,871b83910e04a8a9,f6bfb84d2eea17c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:58:07.181 INFO 4486 --- [host-check-agent-status4,871b83910e04a8a9,f6bfb84d2eea17c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:58:07.182 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:58:07.182 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:58:07.182 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:58:07.183 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:58:07.177+08:00 and status AVAILABLE 2024-10-16 13:58:07.184 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:58:07.185 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.187 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:58:07.190 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:58:07.191 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:07.192 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:07.192 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:07.193 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:07.193 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:07.195 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:07.197 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:07.199 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:07.199 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.200 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:07.213 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.214 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:07.218 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:07.219 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:07.220 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:07.223 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:07.228 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.229 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:07.232 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.233 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:58:07.243 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:07.245 INFO 4486 --- [ocp-schedules-25,2c651cf9ec279254,7eb98ffddbde5d73] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:58:07.252 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:58:07.253 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:58:07.262 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:58:07.264 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:58:07.265 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:58:07.274 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:58:07.275 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:58:07.282 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:07.282 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:07.285 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:58:07.294 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:07.301 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:07.312 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:58:07.321 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:58:07.324 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:58:07.334 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:58:07.335 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:58:07.338 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:07.350 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:58:07.352 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:07.356 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:58:07.358 INFO 4486 --- [ocp-async-6,d97dbc3f4cc5e851,a2db35895b7593a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:07.369 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:07.369 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:58:07.371 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:58:07.380 INFO 4486 --- [host-check-host-status3,92e70bf3ae64ac0c,ffb5c557cc0d740b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:58:07.385 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:58:07.386 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:07.397 INFO 4486 --- [host-check-host-status4,92e70bf3ae64ac0c,ebf9ab32e98a9f72] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:58:07.408 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:58:07.419 INFO 4486 --- [host-check-host-status2,92e70bf3ae64ac0c,ccfd787f6744163d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:58:07.423 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:07.453 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:58:07.464 INFO 4486 --- [host-check-host-status1,92e70bf3ae64ac0c,232a3c0d327c0d00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:58:07.844 INFO 4486 --- [ocp-schedules-15,65c161e38f63e8fc,89aec0d7f2143e1f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:58:07.889 WARN 4486 --- [scanner-13311570-1,65c161e38f63e8fc,6daf490dc27c2a05] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:58:07.994 INFO 4486 --- [prometheus-minute-schedule-1,7949aa1a0b66fb01,98ea3d0c71b9af17] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:08.007 INFO 4486 --- [prometheus-minute-schedule-1,7949aa1a0b66fb01,98ea3d0c71b9af17] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:08.227 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0735dfd69a96524c,26977455ec582f0a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0735dfd69a96524c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:08.227 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0735dfd69a96524c,26977455ec582f0a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0735dfd69a96524c, duration=2 ms] 2024-10-16 13:58:09.358 INFO 4486 --- [ocp-async-6,e394817fc7f30e9f,55aca3c5555143d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:10.006 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:58:10.006 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:58:10.026 INFO 4486 --- [Check obproxy1,08269ea89721f3b9,41fcf7314a8c2239] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:58:10.027 INFO 4486 --- [Check obproxy2,7d24038cfb9ef973,793adf45e837d9c3] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:58:10.027 INFO 4486 --- [Check obproxy1,08269ea89721f3b9,41fcf7314a8c2239] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:10.027 INFO 4486 --- [Check obproxy2,7d24038cfb9ef973,793adf45e837d9c3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:10.029 INFO 4486 --- [Collect obproxy1,c40cb9772eec379f,e308c69444c0494f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:58:10.029 INFO 4486 --- [Collect obproxy1,c40cb9772eec379f,e308c69444c0494f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:10.030 INFO 4486 --- [Collect obproxy2,86efcaab171a6367,310dd5bc3c812649] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:58:10.030 INFO 4486 --- [Collect obproxy2,86efcaab171a6367,310dd5bc3c812649] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:10.049 INFO 4486 --- [Check obproxy2,7d24038cfb9ef973,793adf45e837d9c3] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:58:10.049 INFO 4486 --- [Check obproxy2,7d24038cfb9ef973,793adf45e837d9c3] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:58:10.050 INFO 4486 --- [Collect obproxy2,86efcaab171a6367,310dd5bc3c812649] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:58:10.052 INFO 4486 --- [Check obproxy1,08269ea89721f3b9,41fcf7314a8c2239] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:58:10.053 INFO 4486 --- [Check obproxy1,08269ea89721f3b9,41fcf7314a8c2239] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:58:10.055 INFO 4486 --- [Collect obproxy2,86efcaab171a6367,310dd5bc3c812649] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:58:10.061 INFO 4486 --- [Collect obproxy2,86efcaab171a6367,310dd5bc3c812649] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:58:10.064 INFO 4486 --- [Collect obproxy1,c40cb9772eec379f,e308c69444c0494f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:58:10.067 INFO 4486 --- [Collect obproxy1,c40cb9772eec379f,e308c69444c0494f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:58:10.070 INFO 4486 --- [Collect obproxy1,c40cb9772eec379f,e308c69444c0494f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:58:10.281 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5cc264fb7f0747de,6ed506ccfbe85596] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5cc264fb7f0747de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:10.281 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5cc264fb7f0747de,6ed506ccfbe85596] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5cc264fb7f0747de, duration=1 ms] 2024-10-16 13:58:10.423 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:58:10.994 INFO 4486 --- [prometheus-minute-schedule-1,6b7fdd367bd89c1e,435c214bb6f96cad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:11.001 INFO 4486 --- [prometheus-minute-schedule-1,6b7fdd367bd89c1e,435c214bb6f96cad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:58:11.160 INFO 4486 --- [ocp-schedules-8,b791ff1a5121a6ea,e7754d5189bc26b1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:58:11.358 INFO 4486 --- [ocp-async-2,48e354c00c69c9c8,6af1b6c0ba41cde1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:11.812 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7249e6814401a8b0,991d56352ec7d480] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7249e6814401a8b0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:11.812 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7249e6814401a8b0,991d56352ec7d480] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7249e6814401a8b0, duration=1 ms] 2024-10-16 13:58:12.143 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d4b8fb9896df9090,2afde57f042be573] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d4b8fb9896df9090, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:12.143 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d4b8fb9896df9090,2afde57f042be573] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d4b8fb9896df9090, duration=1 ms] 2024-10-16 13:58:13.023 INFO 4486 --- [ocp-async-7,c48467a39423087c,175d0dc127c52a87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:58:13.359 INFO 4486 --- [ocp-async-4,6f3291e733d63169,6006870aaef3368e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:13.424 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:13.994 INFO 4486 --- [prometheus-minute-schedule-1,4a4795d37e1fc84b,95bb6440eb992025] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:13.999 INFO 4486 --- [prometheus-minute-schedule-1,4a4795d37e1fc84b,95bb6440eb992025] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:14.399 ERROR 4486 --- [metric-archive-second-27,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '587073990944165888-1729058220' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:58:14.397808] [YB420AC531B8-00060B7DB7688655-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '587073990944165888-1729058220' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:58:14.397808] [YB420AC531B8-00060B7DB7688655-0-0] 2024-10-16 13:58:14.436 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058220, nodeCount=1000, affectRows=1000 2024-10-16 13:58:14.439 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058220, nodeCount=1000, affectRows=1000 2024-10-16 13:58:14.441 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058220, nodeCount=1000, validCount=1000 2024-10-16 13:58:14.443 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058220, nodeCount=999, validCount=999 2024-10-16 13:58:14.457 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058220, nodeCount=1000, affectRows=1000 2024-10-16 13:58:14.461 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058220, nodeCount=999, validCount=999 2024-10-16 13:58:14.666 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058220, nodeCount=451, affectRows=451 2024-10-16 13:58:14.667 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058220, nodeCount=450, validCount=450 2024-10-16 13:58:15.176 INFO 4486 --- [metric-archive-async-22,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058220, nodeCount=263, affectRows=263 2024-10-16 13:58:15.177 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058220, nodeCount=214, affectRows=214 2024-10-16 13:58:15.359 INFO 4486 --- [ocp-async-3,8efdeb48cd311e88,8267c750e810ae7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:15.484 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058220, nodeCount=214, validCount=214 2024-10-16 13:58:15.574 INFO 4486 --- [metric-archive-async-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058220, nodeCount=263, validCount=263 2024-10-16 13:58:16.425 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:16.994 INFO 4486 --- [prometheus-minute-schedule-1,f4be80360606a557,c00254eeea642f77] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:17.001 INFO 4486 --- [prometheus-minute-schedule-1,f4be80360606a557,c00254eeea642f77] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:17.069 INFO 4486 --- [ocp-schedules-29,48bf592a2a1b27ca,c16030ca6ccaf9b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:58:17.183 INFO 4486 --- [ocp-schedules-22,4e81fd7615995859,fd81417f0dce7a6b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:58:17.258 INFO 4486 --- [ocp-schedules-2,32188ebfaeb6f7d8,4df4f5e15f316a06] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:58:17.359 INFO 4486 --- [ocp-async-7,624b2c34618ea435,3f7510986275a2e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:17.806 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,54c06d745b7be6d1,8e8c18fa6545f7b8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=54c06d745b7be6d1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:58:17.806 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,54c06d745b7be6d1,8e8c18fa6545f7b8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=54c06d745b7be6d1, duration=1 ms] 2024-10-16 13:58:18.172 INFO 4486 --- [ocp-schedules-6,d5ba7f4c200d6640,1dce87b066a1594e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:58:18.205 WARN 4486 --- [scanner-13312612-1,d5ba7f4c200d6640,063c4c802167ca2a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:58:18.232 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7f9bba94443387e1,56f6e3e4b6e8c8ea] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7f9bba94443387e1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:18.232 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7f9bba94443387e1,56f6e3e4b6e8c8ea] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7f9bba94443387e1, duration=1 ms] 2024-10-16 13:58:18.259 INFO 4486 --- [alarm-detect-13,d5ba7f4c200d6640,964230ebba849fcb] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058265 ~ 1729058288 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058280, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058280, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058280=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:58:19.216 INFO 4486 --- [ocp-schedules-3,683af2f1ad203cbc,7a39b69343cda45e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:58:19.359 INFO 4486 --- [ocp-async-10,8afb17e42876006b,a3faddf1175d297b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:19.426 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:19.994 INFO 4486 --- [prometheus-minute-schedule-1,6ca9118e92381ddc,59817d6a853340f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:20.001 INFO 4486 --- [prometheus-minute-schedule-1,6ca9118e92381ddc,59817d6a853340f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:20.285 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0073f78464c4235d,73397b649978617a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0073f78464c4235d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:20.286 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0073f78464c4235d,73397b649978617a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0073f78464c4235d, duration=2 ms] 2024-10-16 13:58:20.813 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 13:58:20.996 INFO 4486 --- [prometheus-cache-monitor1,d5041a62d8958b10,630320a8b94a5477] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8468} 2024-10-16 13:58:20.996 INFO 4486 --- [prometheus-cache-monitor1,d5041a62d8958b10,630320a8b94a5477] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:58:21.166 INFO 4486 --- [ocp-schedules-8,2b2400562654bb49,e895cba3f0da1c13] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:58:21.360 INFO 4486 --- [ocp-async-9,7cf3a7c97a1d383a,c1fe6c8b05e0f128] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:21.817 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d45b3cfef31b4239,55c03a4820f2a7b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d45b3cfef31b4239, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:21.817 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d45b3cfef31b4239,55c03a4820f2a7b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d45b3cfef31b4239, duration=2 ms] 2024-10-16 13:58:22.147 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,25ad3c0315b493d3,e774b7222f23f8ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=25ad3c0315b493d3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:22.147 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,25ad3c0315b493d3,e774b7222f23f8ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=25ad3c0315b493d3, duration=1 ms] 2024-10-16 13:58:22.428 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:22.994 INFO 4486 --- [prometheus-minute-schedule-1,80675953ecaa58e5,8af5df01fabe5c73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:23.001 INFO 4486 --- [prometheus-minute-schedule-1,80675953ecaa58e5,8af5df01fabe5c73] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:23.024 INFO 4486 --- [ocp-async-1,d03d71b665bd4999,04f2f8f5b1cec26e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:58:23.360 INFO 4486 --- [ocp-async-2,262ad6da8fc338e4,746f77e19ab78d8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:23.827 INFO 4486 --- [ocp-schedules-26,f5d1cf6412e8094c,e44b531211e6aee8] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:58:25.112 INFO 4486 --- [ocp-schedules-24,2146397c34bcb05f,bca8b8f21f8e00f2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:58:25.360 INFO 4486 --- [ocp-async-8,ec97fa26df456b4d,9c81c213222d98ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:25.428 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:25.994 INFO 4486 --- [prometheus-minute-schedule-1,6f3aed6e62cb98b1,ce5bd3dc8cfc3242] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:26.000 INFO 4486 --- [prometheus-minute-schedule-1,6f3aed6e62cb98b1,ce5bd3dc8cfc3242] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:26.540 INFO 4486 --- [ocp-schedules-15,846fb66e6a8b95e8,39847b48800992e3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:58:26.540 INFO 4486 --- [ocp-schedules-15,846fb66e6a8b95e8,39847b48800992e3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:58:26.576 INFO 4486 --- [Collect all tenants major compactions2,846fb66e6a8b95e8,5d875da6c3b40bc4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:58:26.576 INFO 4486 --- [Collect all tenants major compactions3,846fb66e6a8b95e8,a30278782651ee41] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:58:26.577 INFO 4486 --- [Collect all tenants major compactions1,846fb66e6a8b95e8,f40c37a7bc88db61] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:58:26.577 INFO 4486 --- [Collect all tenants major compactions5,846fb66e6a8b95e8,7c0debd906ad98cb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:58:26.577 INFO 4486 --- [Collect all tenants major compactions4,846fb66e6a8b95e8,18b8ad7267cfacd8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:58:26.588 INFO 4486 --- [Collect all tenants major compactions2,846fb66e6a8b95e8,5d875da6c3b40bc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:26.588 INFO 4486 --- [Collect all tenants major compactions1,846fb66e6a8b95e8,f40c37a7bc88db61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:26.589 INFO 4486 --- [Collect all tenants major compactions2,846fb66e6a8b95e8,5d875da6c3b40bc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:58:26.591 INFO 4486 --- [Collect all tenants major compactions1,846fb66e6a8b95e8,f40c37a7bc88db61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:58:26.593 INFO 4486 --- [Collect all tenants major compactions2,846fb66e6a8b95e8,5d875da6c3b40bc4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:58:26.593 INFO 4486 --- [Collect all tenants major compactions2,846fb66e6a8b95e8,5d875da6c3b40bc4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:58:26.594 INFO 4486 --- [Collect all tenants major compactions1,846fb66e6a8b95e8,f40c37a7bc88db61] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:58:26.594 INFO 4486 --- [Collect all tenants major compactions1,846fb66e6a8b95e8,f40c37a7bc88db61] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:58:26.594 INFO 4486 --- [Collect all tenants major compactions3,846fb66e6a8b95e8,a30278782651ee41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:26.596 INFO 4486 --- [Collect all tenants major compactions3,846fb66e6a8b95e8,a30278782651ee41] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:58:26.599 INFO 4486 --- [Collect all tenants major compactions3,846fb66e6a8b95e8,a30278782651ee41] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:58:26.600 INFO 4486 --- [Collect all tenants major compactions3,846fb66e6a8b95e8,a30278782651ee41] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:58:26.605 INFO 4486 --- [Collect all tenants major compactions5,846fb66e6a8b95e8,7c0debd906ad98cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:26.605 INFO 4486 --- [Collect all tenants major compactions4,846fb66e6a8b95e8,18b8ad7267cfacd8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:26.607 INFO 4486 --- [Collect all tenants major compactions5,846fb66e6a8b95e8,7c0debd906ad98cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:58:26.607 INFO 4486 --- [Collect all tenants major compactions4,846fb66e6a8b95e8,18b8ad7267cfacd8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:58:26.611 INFO 4486 --- [Collect all tenants major compactions4,846fb66e6a8b95e8,18b8ad7267cfacd8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:58:26.611 INFO 4486 --- [Collect all tenants major compactions5,846fb66e6a8b95e8,7c0debd906ad98cb] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:58:26.612 INFO 4486 --- [Collect all tenants major compactions4,846fb66e6a8b95e8,18b8ad7267cfacd8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:58:26.612 INFO 4486 --- [Collect all tenants major compactions5,846fb66e6a8b95e8,7c0debd906ad98cb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:58:26.613 INFO 4486 --- [ocp-schedules-15,846fb66e6a8b95e8,39847b48800992e3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:58:27.360 INFO 4486 --- [ocp-async-1,13cc7786af8b5ea3,91733bb20359c293] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:27.764 INFO 4486 --- [ocp-schedules-23,aa5389b4fe525ca2,901f398ba83ff414] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:58:27.764+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:58:27.766 INFO 4486 --- [ocp-schedules-23,aa5389b4fe525ca2,901f398ba83ff414] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:58:27.768 INFO 4486 --- [ocp-schedules-23,aa5389b4fe525ca2,901f398ba83ff414] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:58:27.768 INFO 4486 --- [ocp-schedules-23,aa5389b4fe525ca2,901f398ba83ff414] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:58:28.237 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,45adac32494f90b2,015da916abf9c87a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=45adac32494f90b2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:28.237 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,45adac32494f90b2,015da916abf9c87a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=45adac32494f90b2, duration=1 ms] 2024-10-16 13:58:28.335 INFO 4486 --- [ocp-schedules-25,0e221175e116a3a0,8903e8279c1ad3cf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:58:28.373 INFO 4486 --- [ocp-schedules-11,4b153fa4119c0be8,694d2dee5029aaaf] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 13:58:28.376 INFO 4486 --- [ocp-schedules-11,4b153fa4119c0be8,694d2dee5029aaaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:28.405 WARN 4486 --- [scanner-13312954-1,0e221175e116a3a0,1d3a6f094042aaf3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:58:28.429 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:28.446 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:58:28.447 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.452 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:58:28.463 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:58:28.460 INFO 4486 --- [alarm-detect-8,0e221175e116a3a0,845db0a0260c435c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058280 ~ 1729058298 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058295, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058295, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058295=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:58:28.464 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.465 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.466 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:58:28.469 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:58:28.477 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.478 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:58:28.486 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.487 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:58:28.490 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.491 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:58:28.502 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.503 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:58:28.503 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.504 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:58:28.516 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.516 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.516 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:28.517 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:28.527 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.527 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.528 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:28.528 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:28.534 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:58:28.535 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.538 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.539 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:58:28.539 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 13:58:28.540 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.541 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:58:28.550 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.551 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:58:28.552 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.552 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 13:58:28.553 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.554 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:58:28.567 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.568 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:58:28.573 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.574 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.575 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.575 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.576 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.576 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.577 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.577 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.578 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.579 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.579 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.579 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.580 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 13:58:28.581 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.582 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.583 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.583 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.584 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.584 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.585 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:58:28.585 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.586 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.587 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.588 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.588 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.588 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.589 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.589 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.590 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.590 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.591 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.592 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.592 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.593 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.593 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 13:58:28.593 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.594 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.595 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.596 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:58:28.596 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:58:28.598 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.598 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.602 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.604 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.605 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.605 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 13:58:28.608 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.608 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.609 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:28.609 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.612 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.613 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.613 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:28.614 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.615 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:58:28.616 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.617 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.618 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.619 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:28.620 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 13:58:28.622 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.622 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 13:58:28.622 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.622 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 13:58:28.622 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.622 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 13:58:28.622 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.623 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 13:58:28.623 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:58:28.626 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 13:58:28.626 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:58:28.629 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.630 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:28.648 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:28.652 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.653 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:58:28.659 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.660 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.663 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.663 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.663 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 13:58:28.663 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.663 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 13:58:28.663 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.663 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 13:58:28.665 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.666 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 13:58:28.666 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 13:58:28.667 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.669 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.671 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.672 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.675 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.676 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.679 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.680 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.682 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.683 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 13:58:28.686 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.687 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 13:58:28.690 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.691 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 13:58:28.693 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 13:58:28.695 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 13:58:28.749 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:28.751 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:28.752 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 13:58:28.762 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.762 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 13:58:28.762 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.762 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 13:58:28.762 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 13:58:28.762 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 13:58:28.765 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 13:58:28.765 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:58:28.994 INFO 4486 --- [prometheus-minute-schedule-1,7e4eaf2e2527cf99,5a8176faedc91e2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:29.000 INFO 4486 --- [prometheus-minute-schedule-1,7e4eaf2e2527cf99,5a8176faedc91e2e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:29.361 INFO 4486 --- [ocp-async-6,4287ba4430f1db0d,b94683983a0eb2f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:30.103 INFO 4486 --- [prometheus-second-schedule-1,78f951c29cb02d68,b5970794ab6ab8f3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:58:30.290 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c7cc2e23a8c73030,28b490f8910f8c48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c7cc2e23a8c73030, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:30.290 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c7cc2e23a8c73030,28b490f8910f8c48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c7cc2e23a8c73030, duration=1 ms] 2024-10-16 13:58:31.171 INFO 4486 --- [ocp-schedules-9,e69ce1ad15826929,fab9d4f3871bd654] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:58:31.361 INFO 4486 --- [ocp-async-10,5d0d70ae4f59a711,770b678e7c2bf481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:31.431 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:31.822 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9e90cb75ca676d58,28111050a62b2047] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9e90cb75ca676d58, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:31.822 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,9e90cb75ca676d58,28111050a62b2047] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9e90cb75ca676d58, duration=1 ms] 2024-10-16 13:58:31.994 INFO 4486 --- [prometheus-minute-schedule-1,a022b7fc37ca5c8d,b993f06bfdb1240b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:32.000 INFO 4486 --- [prometheus-minute-schedule-1,a022b7fc37ca5c8d,b993f06bfdb1240b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:32.152 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,459711bcfca140db,b4132310f98b9a46] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=459711bcfca140db, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:32.152 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,459711bcfca140db,b4132310f98b9a46] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=459711bcfca140db, duration=1 ms] 2024-10-16 13:58:33.024 INFO 4486 --- [ocp-async-2,1a5111fc77038a20,a60a5e9f5d02345b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:58:33.361 INFO 4486 --- [ocp-async-1,b31e826f8804f935,126cec9ac3bd48d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:33.755 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6ed2b790bcd8d316,b91cd83e326679c2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=6ed2b790bcd8d316, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:58:33.755 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6ed2b790bcd8d316,b91cd83e326679c2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=6ed2b790bcd8d316, duration=2 ms] 2024-10-16 13:58:34.431 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:34.994 INFO 4486 --- [prometheus-minute-schedule-1,d73d25d933000d13,b4332fd4ecc3617f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:35.000 INFO 4486 --- [prometheus-minute-schedule-1,d73d25d933000d13,b4332fd4ecc3617f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:35.357 INFO 4486 --- [ocp-schedules-16,c6b837e24a8a2466,e78e926cb32f336d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:58:35.357 INFO 4486 --- [ocp-schedules-16,0854cc602ef77b9b,cea24da10d70d183] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:58:35.361 INFO 4486 --- [ocp-schedules-16,0854cc602ef77b9b,cea24da10d70d183] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:35.361 INFO 4486 --- [ocp-async-6,f240bdc773e87cd2,0e4015cb4ed7cd73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:35.391 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.392 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:35.402 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.403 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:35.428 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:35.428 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.428 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.429 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:35.429 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:58:35.461 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:35.462 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.462 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:35.463 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:58:35.464 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.466 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:58:35.468 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:35.476 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.476 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:35.485 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.493 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:35.493 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.494 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:35.494 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:58:35.500 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:58:35.504 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:35.506 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:35.527 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:35.530 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:58:35.530 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.531 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:58:35.554 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:58:35.554 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:58:35.583 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:35.584 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.585 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:58:35.588 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.591 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:58:35.595 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:35.611 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:58:35.621 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:35.622 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.622 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:58:35.623 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.623 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:35.631 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:58:35.631 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:58:35.651 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.651 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:35.674 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.674 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:35.705 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.705 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:58:35.732 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.732 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:58:35.761 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:58:35.786 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:35.788 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:35.789 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:35.794 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:58:35.799 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:58:35.804 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:58:35.807 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:58:35.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:58:35.978 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:58:36.003 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:36.005 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:36.006 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:58:36.032 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:58:36.032 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:58:37.063 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:58:37.064 INFO 4486 --- [ocp-schedules-5,b615b3d07165b936,085880fe7bc06aa8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:58:37.066 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:58:37.066 INFO 4486 --- [ocp-schedules-10,b4512b724aa9e73d,83bf20be142ebca8] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:58:37.067 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:58:37.068 INFO 4486 --- [ocp-schedules-10,b4512b724aa9e73d,83bf20be142ebca8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:37.069 INFO 4486 --- [ocp-schedules-5,b615b3d07165b936,085880fe7bc06aa8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:58:37.081 INFO 4486 --- [ocp-schedules-9,c553ebd9f327ee52,8805bad364e9194c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:58:37.083 INFO 4486 --- [ocp-schedules-9,c553ebd9f327ee52,8805bad364e9194c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729057740308000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:58:37.088 INFO 4486 --- [ocp-schedules-15,b1e80970b4972651,b56a8a27c11e2498] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:58:37.089 INFO 4486 --- [ocp-schedules-29,c7405fe060022601,9a338a81cc0590f6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:58:37.089 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:58:37.089 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:58:37.089 INFO 4486 --- [host-check-agent-status1,c7405fe060022601,68786d5c956ef6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:58:37.090 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:58:37.090 INFO 4486 --- [host-check-agent-status2,c7405fe060022601,29dec476bae20225] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:58:37.090 INFO 4486 --- [ocp-schedules-13,9845cc2648a0db96,eb15eee2546b86e9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:58:37.091 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:58:37.091 INFO 4486 --- [host-check-agent-status3,c7405fe060022601,21b00cdd97dc5af8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:58:37.091 INFO 4486 --- [host-check-agent-status4,c7405fe060022601,b13580f39516dec5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:58:37.092 INFO 4486 --- [host-check-clockdiff2,9845cc2648a0db96,56d88e2bb9290230] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:58:37.092 INFO 4486 --- [host-check-clockdiff1,9845cc2648a0db96,d8a4a73c7a6724d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:58:37.093 INFO 4486 --- [host-check-clockdiff3,9845cc2648a0db96,ff7d4d18eba954e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:58:37.093 INFO 4486 --- [host-check-clockdiff4,9845cc2648a0db96,65a63a124b964997] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:58:37.096 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:37.097 INFO 4486 --- [host-check-agent-status1,c7405fe060022601,68786d5c956ef6a0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:37.097 INFO 4486 --- [host-check-agent-status2,c7405fe060022601,29dec476bae20225] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:37.097 INFO 4486 --- [host-check-agent-status4,c7405fe060022601,b13580f39516dec5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:37.097 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:37.097 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:37.098 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:37.099 INFO 4486 --- [host-check-agent-status1,c7405fe060022601,68786d5c956ef6a0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:37.099 INFO 4486 --- [host-check-agent-status2,c7405fe060022601,29dec476bae20225] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:37.099 INFO 4486 --- [host-check-agent-status4,c7405fe060022601,b13580f39516dec5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:37.099 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:37.099 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:37.099 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:37.099 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:37.100 INFO 4486 --- [host-check-agent-status3,c7405fe060022601,21b00cdd97dc5af8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:37.101 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:37.108 INFO 4486 --- [host-check-agent-status3,c7405fe060022601,21b00cdd97dc5af8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:37.130 INFO 4486 --- [host-check-clockdiff1,9845cc2648a0db96,d8a4a73c7a6724d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058317 0 0 , error= 2024-10-16 13:58:37.135 INFO 4486 --- [host-check-clockdiff4,9845cc2648a0db96,65a63a124b964997] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058317 4 4 , error= 2024-10-16 13:58:37.135 INFO 4486 --- [host-check-clockdiff3,9845cc2648a0db96,ff7d4d18eba954e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058317 5 5 , error= 2024-10-16 13:58:37.135 INFO 4486 --- [host-check-clockdiff2,9845cc2648a0db96,56d88e2bb9290230] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058317 5 5 , error= 2024-10-16 13:58:37.139 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:37.140 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:58:37.154 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:37.155 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:37.156 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.157 INFO 4486 --- [host-check-agent-status1,c7405fe060022601,68786d5c956ef6a0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:58:37.157 INFO 4486 --- [host-check-agent-status2,c7405fe060022601,29dec476bae20225] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:58:37.157 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:58:37.158 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.159 INFO 4486 --- [host-check-agent-status4,c7405fe060022601,b13580f39516dec5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:58:37.159 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:58:37.159 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:58:37.160 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:58:37.160 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:58:37.164247503+08:00 2024-10-16 13:58:37.160 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.160 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:58:37.161 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:58:37.160+08:00 2024-10-16 13:58:37.161 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:58:37.161 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:58:37.165496510+08:00 2024-10-16 13:58:37.161 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:58:37.162 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:58:37.166063580+08:00 2024-10-16 13:58:37.162 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.162 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:58:37.161+08:00 2024-10-16 13:58:37.162 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:58:37.162+08:00 2024-10-16 13:58:37.162 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.163 INFO 4486 --- [host-check-agent-status1,c7405fe060022601,68786d5c956ef6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:58:37.163 INFO 4486 --- [host-check-agent-status1,c7405fe060022601,68786d5c956ef6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:58:37.163 INFO 4486 --- [host-check-agent-status1,c7405fe060022601,68786d5c956ef6a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:58:37.163 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:58:37.160+08:00 and status AVAILABLE 2024-10-16 13:58:37.163 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:58:37.163 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:58:37.163375083+08:00 2024-10-16 13:58:37.164 INFO 4486 --- [host-check-agent-status2,c7405fe060022601,29dec476bae20225] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:58:37.164 INFO 4486 --- [host-check-agent-status2,c7405fe060022601,29dec476bae20225] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:58:37.164 INFO 4486 --- [host-check-agent-status2,c7405fe060022601,29dec476bae20225] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:58:37.164 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:58:37.163+08:00 2024-10-16 13:58:37.164 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:58:37.161+08:00 and status AVAILABLE 2024-10-16 13:58:37.165 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:58:37.162+08:00 and status AVAILABLE 2024-10-16 13:58:37.165 INFO 4486 --- [host-check-agent-status3,c7405fe060022601,21b00cdd97dc5af8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:58:37.166 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:58:37.166 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:58:37.166 INFO 4486 --- [host-check-agent-status4,c7405fe060022601,b13580f39516dec5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:58:37.167 INFO 4486 --- [host-check-agent-status4,c7405fe060022601,b13580f39516dec5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:58:37.167 INFO 4486 --- [host-check-agent-status4,c7405fe060022601,b13580f39516dec5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:58:37.167 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:58:37.167 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:58:37.167 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:58:37.163+08:00 and status AVAILABLE 2024-10-16 13:58:37.168 INFO 4486 --- [host-check-agent-status3,c7405fe060022601,21b00cdd97dc5af8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:58:37.168 INFO 4486 --- [host-check-agent-status3,c7405fe060022601,21b00cdd97dc5af8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:58:37.168 INFO 4486 --- [host-check-agent-status3,c7405fe060022601,21b00cdd97dc5af8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:58:37.170 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:58:37.170 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.171 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:37.173 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.173 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.173 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.173 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.173 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.173 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.175 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:37.175 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:37.176 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:37.176 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:37.177 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:37.177 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:37.177 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:37.178 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:37.181 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.182 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:37.191 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:58:37.192 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.192 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:58:37.194 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.196 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:58:37.198 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:37.199 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:37.201 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:37.203 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:58:37.205 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.206 INFO 4486 --- [ocp-schedules-20,7fc24373e88eaf09,2dac4a93d4003d09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:58:37.231 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:58:37.236 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:58:37.237 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:58:37.242 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:58:37.247 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:58:37.248 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:58:37.259 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:58:37.261 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:37.265 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:37.267 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:37.268 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:58:37.284 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:58:37.291 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:58:37.301 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:58:37.304 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:58:37.313 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:58:37.314 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:58:37.325 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:58:37.325 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:37.331 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:37.340 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:37.342 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:58:37.353 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:58:37.353 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:58:37.361 INFO 4486 --- [ocp-async-4,7a975a57c870bfaf,c0a29f66bb5b3286] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:37.363 INFO 4486 --- [host-check-host-status3,b1e80970b4972651,f419c4a86220a1eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:58:37.367 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:58:37.370 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:58:37.376 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:58:37.378 INFO 4486 --- [host-check-host-status4,b1e80970b4972651,a86c0da1774b2b8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:58:37.389 INFO 4486 --- [host-check-host-status2,b1e80970b4972651,39a913730e995b14] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:58:37.399 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:37.402 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.403 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:58:37.404 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.405 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.406 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:58:37.415 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.415 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.415 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.415 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.415 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.415 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.432 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:58:37.432 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:37.443 INFO 4486 --- [host-check-host-status1,b1e80970b4972651,c28da9d38998ca80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:58:37.445 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:58:37.478 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:58:37.527 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:37.543 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:37.545 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.546 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:58:37.547 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:58:37.548 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.549 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:58:37.550 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.550 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.551 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:58:37.561 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.561 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.561 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.561 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.561 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.561 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.569 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:58:37.761 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:58:37.806 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:37.808 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.809 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:58:37.810 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.811 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.812 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:58:37.823 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.823 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.823 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.823 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.823 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:37.824 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:37.842 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:58:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:58:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:58:37.994 INFO 4486 --- [prometheus-minute-schedule-1,8823c039102af892,ec6e47677b013246] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:37.999 INFO 4486 --- [prometheus-minute-schedule-1,8823c039102af892,ec6e47677b013246] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:38.041 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:58:38.086 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:58:38.088 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:38.089 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:58:38.090 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:38.091 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:38.092 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:58:38.101 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:38.101 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:38.101 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:38.101 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:38.101 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:58:38.102 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:58:38.162 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:58:38.243 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a71b5dc85b51de97,775496fc6f80996b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a71b5dc85b51de97, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:38.243 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a71b5dc85b51de97,775496fc6f80996b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a71b5dc85b51de97, duration=2 ms] 2024-10-16 13:58:38.348 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:58:38.169357, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:58:38.349 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:58:38.169357, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:58:38.392 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:58:38.169357} } 2024-10-16 13:58:38.450 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:58:38.459 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:58:38.462 INFO 4486 --- [ocp-schedules-25,42ea4beca57a8128,1deb5295629db812] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:58:38.530 INFO 4486 --- [ocp-schedules-20,bb77aff47f9deea9,aa1039d81712f7b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:58:38.539 WARN 4486 --- [scanner-13313143-1,bb77aff47f9deea9,0d02ee9599275d43] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:58:39.220 INFO 4486 --- [ocp-schedules-8,0b31838730de4ac5,57b273262b0a192e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:58:39.362 INFO 4486 --- [ocp-async-3,adce9fbdb0511d5a,150434d76f94ba87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:40.007 INFO 4486 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:58:40.007 INFO 4486 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:58:40.026 INFO 4486 --- [Check obproxy1,e4fbb25a288f36a0,f9f86fee6c33866f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:58:40.026 INFO 4486 --- [Check obproxy2,2173eb41e901d80e,dc206c4cf35a7252] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:58:40.026 INFO 4486 --- [Check obproxy1,e4fbb25a288f36a0,f9f86fee6c33866f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:40.026 INFO 4486 --- [Check obproxy2,2173eb41e901d80e,dc206c4cf35a7252] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:40.030 INFO 4486 --- [Collect obproxy1,c739a15a7589b6cd,307f9b5cc89560ae] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:58:40.030 INFO 4486 --- [Collect obproxy2,5f4a1f73919be50d,ba0408f3275d210b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:58:40.030 INFO 4486 --- [Collect obproxy1,c739a15a7589b6cd,307f9b5cc89560ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:40.030 INFO 4486 --- [Collect obproxy2,5f4a1f73919be50d,ba0408f3275d210b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:58:40.050 INFO 4486 --- [Collect obproxy2,5f4a1f73919be50d,ba0408f3275d210b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:58:40.050 INFO 4486 --- [Check obproxy2,2173eb41e901d80e,dc206c4cf35a7252] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:58:40.050 INFO 4486 --- [Check obproxy2,2173eb41e901d80e,dc206c4cf35a7252] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:58:40.055 INFO 4486 --- [Collect obproxy1,c739a15a7589b6cd,307f9b5cc89560ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:58:40.055 INFO 4486 --- [Check obproxy1,e4fbb25a288f36a0,f9f86fee6c33866f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:58:40.055 INFO 4486 --- [Check obproxy1,e4fbb25a288f36a0,f9f86fee6c33866f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:58:40.056 INFO 4486 --- [Collect obproxy2,5f4a1f73919be50d,ba0408f3275d210b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:58:40.058 INFO 4486 --- [Collect obproxy1,c739a15a7589b6cd,307f9b5cc89560ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:58:40.061 INFO 4486 --- [Collect obproxy2,5f4a1f73919be50d,ba0408f3275d210b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:58:40.061 INFO 4486 --- [Collect obproxy1,c739a15a7589b6cd,307f9b5cc89560ae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:58:40.294 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fbb5e1c113b9027a,c3481341911fb8e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fbb5e1c113b9027a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:40.294 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fbb5e1c113b9027a,c3481341911fb8e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fbb5e1c113b9027a, duration=1 ms] 2024-10-16 13:58:40.433 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:40.994 INFO 4486 --- [prometheus-minute-schedule-1,12704a6b32e2fe9e,2818fd36beb8e5aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:40.995 INFO 4486 --- [prometheus-hour-schedule-1,5c5a426f16078287,0b66cdd6793d55e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:58:40.999 INFO 4486 --- [prometheus-minute-schedule-1,12704a6b32e2fe9e,2818fd36beb8e5aa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:41.177 INFO 4486 --- [ocp-schedules-13,c07e3fe861c3a068,990fe699d3c9f6a0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:58:41.362 INFO 4486 --- [ocp-async-7,788ebdc42e82bfe2,9e2b6f0a4a583ada] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:41.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,98e37844e1422f6a,1cfca4a9bb04655f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=98e37844e1422f6a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:41.827 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,98e37844e1422f6a,1cfca4a9bb04655f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=98e37844e1422f6a, duration=1 ms] 2024-10-16 13:58:42.157 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ea3537c91cc8b300,116dd7fa29f3b6f5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ea3537c91cc8b300, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:42.157 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ea3537c91cc8b300,116dd7fa29f3b6f5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ea3537c91cc8b300, duration=1 ms] 2024-10-16 13:58:42.175 INFO 4486 --- [ocp-schedules-8,02b9e4bdfc9adf13,46ca3a59c0d7770b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:58:43.024 INFO 4486 --- [ocp-async-6,54b962a0b59b3767,8aa6562d6aba5d1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:58:43.362 INFO 4486 --- [ocp-async-8,5872674f89b0fe6e,ea24f4a8d518da46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:43.434 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:43.994 INFO 4486 --- [prometheus-minute-schedule-1,f4e9381ea005992a,a61eec40ea382078] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:44.001 INFO 4486 --- [prometheus-minute-schedule-1,f4e9381ea005992a,a61eec40ea382078] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:45.362 INFO 4486 --- [ocp-async-9,2883ea819e3356fc,16ffa36a2d339142] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:45.580 INFO 4486 --- [ocp-schedules-27,379d47764a18eee5,6f8e8f494ff4211b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:58:45.584 INFO 4486 --- [ocp-schedules-27,379d47764a18eee5,6f8e8f494ff4211b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:58:45.584 INFO 4486 --- [ocp-schedules-27,379d47764a18eee5,6f8e8f494ff4211b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:58:45.610 INFO 4486 --- [ocp-schedules-29,8e00c8c69af183eb,f18eb85b14cf1f61] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:58:45.610 INFO 4486 --- [ocp-schedules-29,8e00c8c69af183eb,f18eb85b14cf1f61] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:58:46.435 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:46.994 INFO 4486 --- [prometheus-minute-schedule-1,4be3e470f8bef7a4,a72fd8584fd161f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:47.000 INFO 4486 --- [prometheus-minute-schedule-1,4be3e470f8bef7a4,a72fd8584fd161f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:47.070 INFO 4486 --- [ocp-schedules-10,b4512b724aa9e73d,83bf20be142ebca8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:58:47.167 INFO 4486 --- [ocp-schedules-9,c553ebd9f327ee52,8805bad364e9194c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:58:47.262 INFO 4486 --- [ocp-schedules-24,2d781771ab300ce1,67804661e7f2f76a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:58:47.363 INFO 4486 --- [ocp-async-6,15b21ca93e2d64b5,70acf46e762de863] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:48.286 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c2bc0a64ba8a970b,a5442933b67faf57] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c2bc0a64ba8a970b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:48.286 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c2bc0a64ba8a970b,a5442933b67faf57] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c2bc0a64ba8a970b, duration=2 ms] 2024-10-16 13:58:48.699 INFO 4486 --- [ocp-schedules-22,3d6cf1887a0d5cd4,77befaa231fe86e4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:58:48.728 WARN 4486 --- [scanner-13313498-1,3d6cf1887a0d5cd4,9d5c7f466475bd09] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:58:48.803 INFO 4486 --- [alarm-detect-5,3d6cf1887a0d5cd4,e2078fd01088a39a] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058295 ~ 1729058318 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058310, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058310, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058310=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:58:49.363 INFO 4486 --- [ocp-async-8,2439abf6eeb8a5ce,1630e1004881fa34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:49.436 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:49.994 INFO 4486 --- [prometheus-minute-schedule-1,ce8fffad147257a1,45a4427b846127c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:50.000 INFO 4486 --- [prometheus-minute-schedule-1,ce8fffad147257a1,45a4427b846127c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:50.299 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ecfcd58b32d37382,87e7ec509c295d5e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ecfcd58b32d37382, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:50.300 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ecfcd58b32d37382,87e7ec509c295d5e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ecfcd58b32d37382, duration=2 ms] 2024-10-16 13:58:50.547 INFO 4486 --- [ocp-schedules-6,66a6f3a0737457b5,60a7b4afa56c0837] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:58:50.547 INFO 4486 --- [ocp-schedules-6,66a6f3a0737457b5,60a7b4afa56c0837] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:58:50.548 INFO 4486 --- [ocp-schedules-6,66a6f3a0737457b5,60a7b4afa56c0837] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:58:50.548 INFO 4486 --- [ocp-schedules-6,66a6f3a0737457b5,60a7b4afa56c0837] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:58:50.549 INFO 4486 --- [ocp-schedules-6,66a6f3a0737457b5,60a7b4afa56c0837] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:58:50.549 INFO 4486 --- [ocp-schedules-6,66a6f3a0737457b5,60a7b4afa56c0837] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 13:58:51.183 INFO 4486 --- [ocp-schedules-29,b2ed11247de55f0a,abe2b3910b518e2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:58:51.334 INFO 4486 --- [ocp-schedules-12,751539068e8a6fbb,2b99c5d8fe7ddfec] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:58:51.335 INFO 4486 --- [ocp-schedules-12,751539068e8a6fbb,2b99c5d8fe7ddfec] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:58:51.335 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:58:51.335 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:58:51.335 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:58:51.335 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:58:51.335 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:58:51.341 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:58:51.341 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:58:51.341 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:58:51.342 INFO 4486 --- [ocp-schedules-12,751539068e8a6fbb,2b99c5d8fe7ddfec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:58:51.342 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:58:51.343 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:58:51.343 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:58:51.343 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:58:51.343 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:58:51.343 INFO 4486 --- [ocp-schedules-12,751539068e8a6fbb,2b99c5d8fe7ddfec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:58:51.344 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:58:51.345 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:58:51.363 INFO 4486 --- [ocp-async-3,afab7480ae3e2310,9cf0f28bd6ccce48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:51.370 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:58:51.371 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:58:51.371 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:58:51.372 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:58:51.372 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:58:51.378 INFO 4486 --- [ocp-schedules-12,751539068e8a6fbb,2b99c5d8fe7ddfec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:58:51.381 INFO 4486 --- [ocp-schedules-12,751539068e8a6fbb,2b99c5d8fe7ddfec] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:58:51.831 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a9c565b1549b18f3,9c05db4285572709] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a9c565b1549b18f3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:51.831 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a9c565b1549b18f3,9c05db4285572709] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a9c565b1549b18f3, duration=1 ms] 2024-10-16 13:58:52.161 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,424617193302ddc6,5b29e3bde33e04c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=424617193302ddc6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:52.162 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,424617193302ddc6,5b29e3bde33e04c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=424617193302ddc6, duration=1 ms] 2024-10-16 13:58:52.437 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:52.994 INFO 4486 --- [prometheus-minute-schedule-1,d2ecb7f4c64d2d81,627d3ee0098fcf2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:53.000 INFO 4486 --- [prometheus-minute-schedule-1,d2ecb7f4c64d2d81,627d3ee0098fcf2e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:53.024 INFO 4486 --- [ocp-async-9,e3850fb4b6d56c98,fa52b5fb25042325] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:58:53.363 INFO 4486 --- [ocp-async-6,f356cf0cf153ec39,eb087b82173e1658] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:53.569 INFO 4486 --- [ocp-schedules-9,76f5d2961f45a223,c075f7188ea2b6dc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 13:58:53.572 INFO 4486 --- [ocp-schedules-9,76f5d2961f45a223,c075f7188ea2b6dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:58:53.572 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:58:53.573 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:58:53.573 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 13:58:54.741 INFO 4486 --- [ocp-schedules-6,1ee55d3999902de5,c81a2e603248ca3f] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T13:58:54.741+08:00, tenantSize=5 2024-10-16 13:58:54.746 INFO 4486 --- [ocp-schedules-6,1ee55d3999902de5,c81a2e603248ca3f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729058334741000 2024-10-16 13:58:54.746 INFO 4486 --- [ocp-schedules-6,1ee55d3999902de5,c81a2e603248ca3f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729058034714000 2024-10-16 13:58:54.762 INFO 4486 --- [ocp-schedules-6,1ee55d3999902de5,c81a2e603248ca3f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 13:58:54.768 INFO 4486 --- [ocp-schedules-6,1ee55d3999902de5,c81a2e603248ca3f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729058334765000 2024-10-16 13:58:54.768 INFO 4486 --- [ocp-schedules-6,1ee55d3999902de5,c81a2e603248ca3f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729058034681000 2024-10-16 13:58:54.800 INFO 4486 --- [ocp-schedules-6,1ee55d3999902de5,c81a2e603248ca3f] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 13:58:55.363 INFO 4486 --- [ocp-async-8,143ab771d0dadcff,0802b372441f4495] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:55.437 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:55.994 INFO 4486 --- [prometheus-minute-schedule-1,2c1956c8ee5cf00d,93de51cbf42ceb59] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:56.000 INFO 4486 --- [prometheus-minute-schedule-1,2c1956c8ee5cf00d,93de51cbf42ceb59] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:56.440 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,36173d913cbebb30,76ac70a160ecd776] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=36173d913cbebb30, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:58:56.441 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,36173d913cbebb30,76ac70a160ecd776] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=36173d913cbebb30, duration=2 ms] 2024-10-16 13:58:57.364 INFO 4486 --- [ocp-async-9,ca24d92fa8e09062,1e7fc13a09eef81f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:58:58.290 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b5986f0518cfe334,304d368b68da2bf8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b5986f0518cfe334, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:58:58.290 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b5986f0518cfe334,304d368b68da2bf8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b5986f0518cfe334, duration=1 ms] 2024-10-16 13:58:58.438 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:58:58.861 INFO 4486 --- [ocp-schedules-8,865542ca13b5c06e,df6b2f01301ce47e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:58:58.904 WARN 4486 --- [scanner-13313731-1,865542ca13b5c06e,b22bb1a668cf8046] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:58:58.994 INFO 4486 --- [prometheus-minute-schedule-1,ec290e7dc29f5cbe,40343a3184d5f716] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058340, interval=60 2024-10-16 13:58:59.001 INFO 4486 --- [prometheus-minute-schedule-1,ec290e7dc29f5cbe,40343a3184d5f716] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:58:58.999 INFO 4486 --- [alarm-detect-12,865542ca13b5c06e,0ed76988270a99f0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058310 ~ 1729058328 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058325, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058325, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058325=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:58:59.191 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,71aa27c65fba0145,0d613149669cd5fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=71aa27c65fba0145, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:58:59.192 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,71aa27c65fba0145,0d613149669cd5fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=71aa27c65fba0145, duration=1 ms] 2024-10-16 13:58:59.225 INFO 4486 --- [ocp-schedules-23,92260db5ef503a82,d5feeed476419a62] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:58:59.364 INFO 4486 --- [ocp-async-2,6e4aaf4a59b50558,f32b920b6862b47a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:00.306 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bf1444a9a0361d5,4546ac1a7f52f8ed] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obce_3zones&version=2&ObClusterId=1701313639, client=10.197.49.176, traceId=6bf1444a9a0361d5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obce_3zones,1701313639,2,{"ObRegion":"obce_3zones","ObCluster":"obce_3zones","ObRegionId":1701313639,"ObClusterId":1701313639,"Type":"PRIMARY","timestamp":1729058340307405,"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 13:59:00.309 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ef03ae959a06eedf,2c7023fd289c0eee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ef03ae959a06eedf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:00.309 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ef03ae959a06eedf,2c7023fd289c0eee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ef03ae959a06eedf, duration=1 ms] 2024-10-16 13:59:00.310 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bf1444a9a0361d5,4546ac1a7f52f8ed] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=obce_3zones, obClusterId=1701313639, rootServerJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, groupName=obce_3zones, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 13:59:00.310 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bf1444a9a0361d5,4546ac1a7f52f8ed] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 13:59:00.310 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bf1444a9a0361d5,4546ac1a7f52f8ed] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=obce_3zones, clusterId=1701313639, affectRows=1 2024-10-16 13:59:00.311 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bf1444a9a0361d5,4546ac1a7f52f8ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.176, traceId=6bf1444a9a0361d5, duration=7 ms] 2024-10-16 13:59:01.188 INFO 4486 --- [ocp-schedules-6,da4369bdb627c574,013256969bb138fb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:59:01.364 INFO 4486 --- [ocp-async-10,65ffe4fbc53a4af5,197d08ac899b4458] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:01.439 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:01.835 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ee0a0e96758c09c2,124302aef421fd9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ee0a0e96758c09c2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:01.835 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ee0a0e96758c09c2,124302aef421fd9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ee0a0e96758c09c2, duration=0 ms] 2024-10-16 13:59:01.994 INFO 4486 --- [prometheus-minute-schedule-1,ddd05549a64687e2,543eea74cff9509c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:01.999 INFO 4486 --- [prometheus-minute-schedule-1,ddd05549a64687e2,543eea74cff9509c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:02.165 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3655d1f98dbd75e5,75e8e4f830f50f34] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3655d1f98dbd75e5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:02.165 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3655d1f98dbd75e5,75e8e4f830f50f34] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3655d1f98dbd75e5, duration=1 ms] 2024-10-16 13:59:02.536 INFO 4486 --- [ocp-schedules-25,1691f9e65d2b19e0,0c84c9d8e863df78] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T13:59:02.536+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:59:02.539 INFO 4486 --- [ocp-schedules-25,1691f9e65d2b19e0,0c84c9d8e863df78] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 13:59:02.543 INFO 4486 --- [ocp-schedules-25,1691f9e65d2b19e0,0c84c9d8e863df78] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 13:59:02.556 INFO 4486 --- [ocp-schedules-25,1691f9e65d2b19e0,0c84c9d8e863df78] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 13:59:03.024 INFO 4486 --- [ocp-async-6,62033d4b8d427d68,87b475c243b58a36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:59:03.364 INFO 4486 --- [ocp-async-3,8d0a0d0be4a7ad59,e2222ede6da34376] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:03.573 INFO 4486 --- [ocp-schedules-9,76f5d2961f45a223,c075f7188ea2b6dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:59:04.440 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:04.994 INFO 4486 --- [prometheus-minute-schedule-1,0e8fa1744278550f,e5144d1b4e63d736] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:05.001 INFO 4486 --- [prometheus-minute-schedule-1,0e8fa1744278550f,e5144d1b4e63d736] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:05.364 INFO 4486 --- [ocp-async-2,500393a3ab35d2ca,02eacbb9998d126d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:06.033 INFO 4486 --- [ocp-schedules-16,0854cc602ef77b9b,cea24da10d70d183] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:59:06.038 INFO 4486 --- [ocp-schedules-16,5896ab4dfbbd0459,1086fcae0c8f8c3f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:59:06.041 INFO 4486 --- [ocp-schedules-16,5896ab4dfbbd0459,1086fcae0c8f8c3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:59:06.074 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:06.083 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.083 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:06.174 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.175 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:06.180 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.181 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.181 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:59:06.223 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.224 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.224 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:06.225 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:59:06.225 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.228 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:59:06.229 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:06.239 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.239 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:06.250 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.259 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.260 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.261 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:06.261 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:59:06.266 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:59:06.271 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:06.273 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:06.276 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.278 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.279 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:59:06.298 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:59:06.309 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:59:06.310 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:59:06.327 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.328 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.329 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:06.333 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.336 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:59:06.340 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:06.367 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.367 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:06.382 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:59:06.391 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.392 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.393 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:59:06.394 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.394 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:06.403 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:59:06.403 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:59:06.423 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.423 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:06.451 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.451 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:06.475 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.475 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:06.502 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:06.528 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.530 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.531 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:06.536 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:59:06.543 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:59:06.548 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:59:06.550 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:59:06.572 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:59:06.710 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:59:06.753 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:06.755 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:06.756 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:59:06.795 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:59:06.795 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:59:07.066 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:59:07.066 INFO 4486 --- [ocp-schedules-17,af18210788e24512,dca9c8d834049f42] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:59:07.068 INFO 4486 --- [ocp-schedules-17,af18210788e24512,dca9c8d834049f42] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:59:07.081 INFO 4486 --- [ocp-schedules-30,10586fc94267d411,452626b2ef9b0f60] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:59:07.084 INFO 4486 --- [ocp-schedules-30,10586fc94267d411,452626b2ef9b0f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:59:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:59:07.089 INFO 4486 --- [ocp-schedules-22,fb1d5ae981ee502f,53b8e9f86b7a9d78] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:59:07.089 INFO 4486 --- [ocp-schedules-18,68abd57ce864a7cb,f09b8a70ce546b34] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:59:07.089 INFO 4486 --- [ocp-schedules-6,efd5765b66bd8312,ea4f49f832f29067] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:59:07.089 INFO 4486 --- [host-check-agent-status1,68abd57ce864a7cb,6e0a445e1273609f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:59:07.089 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:59:07.090 INFO 4486 --- [host-check-clockdiff1,fb1d5ae981ee502f,ce14fda3814674ac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:59:07.090 INFO 4486 --- [host-check-clockdiff2,fb1d5ae981ee502f,dad67a65d38c8e81] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:59:07.090 INFO 4486 --- [host-check-clockdiff3,fb1d5ae981ee502f,516e79cbc7f33688] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:59:07.090 INFO 4486 --- [host-check-agent-status2,68abd57ce864a7cb,9c33626cb2b2d7de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:59:07.091 INFO 4486 --- [host-check-agent-status3,68abd57ce864a7cb,c0c8e110e2bffb2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:59:07.092 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:59:07.092 INFO 4486 --- [host-check-agent-status4,68abd57ce864a7cb,42b3a69323c0193d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:59:07.093 INFO 4486 --- [host-check-clockdiff4,fb1d5ae981ee502f,931d52e7290947d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:59:07.093 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:59:07.093 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:59:07.098 INFO 4486 --- [host-check-agent-status1,68abd57ce864a7cb,6e0a445e1273609f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:07.099 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:59:07.099 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:07.099 INFO 4486 --- [host-check-agent-status1,68abd57ce864a7cb,6e0a445e1273609f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:07.110 INFO 4486 --- [host-check-agent-status2,68abd57ce864a7cb,9c33626cb2b2d7de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:59:07.131 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:59:07.135 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:59:07.135 INFO 4486 --- [host-check-agent-status4,68abd57ce864a7cb,42b3a69323c0193d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:59:07.137 INFO 4486 --- [host-check-clockdiff2,fb1d5ae981ee502f,dad67a65d38c8e81] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058347 5 5 , error= 2024-10-16 13:59:07.137 INFO 4486 --- [host-check-clockdiff1,fb1d5ae981ee502f,ce14fda3814674ac] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058347 0 0 , error= 2024-10-16 13:59:07.137 INFO 4486 --- [host-check-agent-status3,68abd57ce864a7cb,c0c8e110e2bffb2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:59:07.137 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:07.138 INFO 4486 --- [host-check-agent-status2,68abd57ce864a7cb,9c33626cb2b2d7de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:59:07.138 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:59:07.139 INFO 4486 --- [host-check-agent-status4,68abd57ce864a7cb,42b3a69323c0193d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:59:07.139 INFO 4486 --- [host-check-agent-status3,68abd57ce864a7cb,c0c8e110e2bffb2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:59:07.140 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:59:07.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.142 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:59:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:07.143 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:59:07.161 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.161 INFO 4486 --- [host-check-agent-status1,68abd57ce864a7cb,6e0a445e1273609f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:59:07.162 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:07.163 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:59:07.164 INFO 4486 --- [host-check-agent-status3,68abd57ce864a7cb,c0c8e110e2bffb2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:59:07.164 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:59:07.166 INFO 4486 --- [host-check-agent-status4,68abd57ce864a7cb,42b3a69323c0193d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:59:07.166 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:59:07.166 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:59:07.169748089+08:00 2024-10-16 13:59:07.166 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:59:07.166 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:59:07.166 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:59:07.170611515+08:00 2024-10-16 13:59:07.166 INFO 4486 --- [host-check-agent-status2,68abd57ce864a7cb,9c33626cb2b2d7de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:59:07.167 INFO 4486 --- [host-check-agent-status1,68abd57ce864a7cb,6e0a445e1273609f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:59:07.167 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:59:07.166+08:00 2024-10-16 13:59:07.167 INFO 4486 --- [host-check-agent-status1,68abd57ce864a7cb,6e0a445e1273609f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:59:07.167 INFO 4486 --- [host-check-agent-status1,68abd57ce864a7cb,6e0a445e1273609f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:59:07.167 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:59:07.166+08:00 2024-10-16 13:59:07.168 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:59:07.168007687+08:00 2024-10-16 13:59:07.168 INFO 4486 --- [host-check-agent-status3,68abd57ce864a7cb,c0c8e110e2bffb2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:59:07.168 INFO 4486 --- [host-check-agent-status3,68abd57ce864a7cb,c0c8e110e2bffb2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:59:07.168 INFO 4486 --- [host-check-agent-status3,68abd57ce864a7cb,c0c8e110e2bffb2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:59:07.168 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:59:07.172442849+08:00 2024-10-16 13:59:07.169 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:59:07.168+08:00 2024-10-16 13:59:07.169 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:59:07.169+08:00 2024-10-16 13:59:07.170 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:59:07.166+08:00 and status AVAILABLE 2024-10-16 13:59:07.170 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:59:07.166+08:00 and status AVAILABLE 2024-10-16 13:59:07.171 INFO 4486 --- [host-check-agent-status4,68abd57ce864a7cb,42b3a69323c0193d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:59:07.171 INFO 4486 --- [host-check-agent-status2,68abd57ce864a7cb,9c33626cb2b2d7de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:59:07.171 INFO 4486 --- [host-check-agent-status4,68abd57ce864a7cb,42b3a69323c0193d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:59:07.171 INFO 4486 --- [host-check-agent-status2,68abd57ce864a7cb,9c33626cb2b2d7de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:59:07.171 INFO 4486 --- [host-check-agent-status2,68abd57ce864a7cb,9c33626cb2b2d7de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:59:07.171 INFO 4486 --- [host-check-agent-status4,68abd57ce864a7cb,42b3a69323c0193d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:59:07.172 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:59:07.169+08:00 and status AVAILABLE 2024-10-16 13:59:07.172 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:59:07.168+08:00 and status AVAILABLE 2024-10-16 13:59:07.172 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:59:07.172 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:59:07.173 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.174 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:59:07.174 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:59:07.177 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:59:07.180 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:59:07.181 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:59:07.181 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:59:07.182 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:59:07.182 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:59:07.183 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:59:07.183 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:07.185 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:07.186 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.187 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:59:07.200 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.201 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:07.201 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:59:07.206 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:07.207 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:07.208 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:07.215 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.216 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:59:07.222 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.224 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:59:07.236 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:07.237 INFO 4486 --- [ocp-schedules-19,ddf071106141c916,95efddb5adb8c7d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:59:07.242 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:59:07.242 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:59:07.243 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:59:07.252 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:59:07.253 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:59:07.255 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:59:07.257 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:59:07.267 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:59:07.269 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:07.269 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:07.271 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:07.282 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:07.298 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:59:07.301 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:59:07.309 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:59:07.311 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:59:07.313 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:59:07.325 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:59:07.326 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:07.326 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:07.342 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:59:07.343 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:07.353 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:59:07.359 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:59:07.360 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:59:07.365 INFO 4486 --- [ocp-async-6,dc294b2923269e01,7303a584c5cb5dfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:07.370 INFO 4486 --- [host-check-host-status4,efd5765b66bd8312,36c11e1be2c8eb93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:59:07.370 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:07.371 INFO 4486 --- [host-check-host-status3,efd5765b66bd8312,720947980256dd5d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:59:07.377 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:59:07.389 INFO 4486 --- [host-check-host-status2,efd5765b66bd8312,dc0269504db75b95] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:59:07.424 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:59:07.436 INFO 4486 --- [host-check-host-status1,efd5765b66bd8312,0a94c3a6da70e017] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:59:07.441 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:07.994 INFO 4486 --- [prometheus-minute-schedule-1,92d6702d265fdb2a,7132a4b97ffa858f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:08.002 INFO 4486 --- [prometheus-minute-schedule-1,92d6702d265fdb2a,7132a4b97ffa858f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:08.048 INFO 4486 --- [host-check-clockdiff3,fb1d5ae981ee502f,516e79cbc7f33688] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058348 5 5 , error= 2024-10-16 13:59:08.296 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,aace69bad7d0167d,f78a61ac1216b0dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=aace69bad7d0167d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:08.296 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,aace69bad7d0167d,f78a61ac1216b0dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=aace69bad7d0167d, duration=2 ms] 2024-10-16 13:59:09.069 INFO 4486 --- [ocp-schedules-6,655f34953d6c94f4,68f9b5c5e4533968] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:59:09.093 WARN 4486 --- [scanner-13314086-1,655f34953d6c94f4,bb2e7b8761c02948] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:59:09.365 INFO 4486 --- [ocp-async-8,2554c6c254721848,6bee67e8eb67b7ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:09.633 INFO 4486 --- [host-check-clockdiff4,fb1d5ae981ee502f,931d52e7290947d4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058349 5 5 , error= 2024-10-16 13:59:10.006 INFO 4486 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:59:10.007 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:59:10.025 INFO 4486 --- [Check obproxy1,915e0bcb86495fc8,9432b708c1f65050] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:59:10.025 INFO 4486 --- [Check obproxy1,915e0bcb86495fc8,9432b708c1f65050] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:10.025 INFO 4486 --- [Check obproxy2,e1ad028bffcd7841,c9f27b43ca3ad488] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:59:10.026 INFO 4486 --- [Check obproxy2,e1ad028bffcd7841,c9f27b43ca3ad488] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:10.030 INFO 4486 --- [Collect obproxy1,26fd8f893ce7ff50,d974015e2929bfe4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:59:10.030 INFO 4486 --- [Collect obproxy1,26fd8f893ce7ff50,d974015e2929bfe4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:10.030 INFO 4486 --- [Collect obproxy2,cc426e60d6e40b3f,a45afcaa095abb14] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:59:10.030 INFO 4486 --- [Collect obproxy2,cc426e60d6e40b3f,a45afcaa095abb14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:10.048 INFO 4486 --- [Check obproxy1,915e0bcb86495fc8,9432b708c1f65050] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:59:10.048 INFO 4486 --- [Check obproxy1,915e0bcb86495fc8,9432b708c1f65050] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:59:10.049 INFO 4486 --- [Check obproxy2,e1ad028bffcd7841,c9f27b43ca3ad488] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:59:10.050 INFO 4486 --- [Check obproxy2,e1ad028bffcd7841,c9f27b43ca3ad488] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:59:10.052 INFO 4486 --- [Collect obproxy2,cc426e60d6e40b3f,a45afcaa095abb14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:59:10.057 INFO 4486 --- [Collect obproxy2,cc426e60d6e40b3f,a45afcaa095abb14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:59:10.059 INFO 4486 --- [Collect obproxy1,26fd8f893ce7ff50,d974015e2929bfe4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:59:10.062 INFO 4486 --- [Collect obproxy2,cc426e60d6e40b3f,a45afcaa095abb14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:59:10.064 INFO 4486 --- [Collect obproxy1,26fd8f893ce7ff50,d974015e2929bfe4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:59:10.068 INFO 4486 --- [Collect obproxy1,26fd8f893ce7ff50,d974015e2929bfe4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:59:10.314 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,b33b4ceb570774bc,39a3cf3e8c4aca7b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b33b4ceb570774bc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:10.315 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,b33b4ceb570774bc,39a3cf3e8c4aca7b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b33b4ceb570774bc, duration=1 ms] 2024-10-16 13:59:10.442 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 13:59:10.994 INFO 4486 --- [prometheus-minute-schedule-1,569f09fed534f213,c4366b8c00c87bb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:11.000 INFO 4486 --- [prometheus-minute-schedule-1,569f09fed534f213,c4366b8c00c87bb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:11.007 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 13:59:11.194 INFO 4486 --- [ocp-schedules-12,17278adafa69b2dc,f4cd098cdeaa1569] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:59:11.365 INFO 4486 --- [ocp-async-3,de078324456f6c43,7d389716209eb299] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:11.852 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bd4ae50c360f0b7,8b539ddaa71407d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6bd4ae50c360f0b7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:11.852 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6bd4ae50c360f0b7,8b539ddaa71407d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6bd4ae50c360f0b7, duration=1 ms] 2024-10-16 13:59:12.170 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2355191d5e1cdc2c,a64a7b2a1382d1f4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2355191d5e1cdc2c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:12.170 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2355191d5e1cdc2c,a64a7b2a1382d1f4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2355191d5e1cdc2c, duration=1 ms] 2024-10-16 13:59:13.025 INFO 4486 --- [ocp-async-1,dae81ab3249c1325,63f175ceba47b314] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:59:13.365 INFO 4486 --- [ocp-async-5,e3b3b7a18bd30aa6,e046a88f3b9230e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:13.442 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:13.994 INFO 4486 --- [prometheus-minute-schedule-1,e4ddcbc479045795,76dbb8f374486156] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:14.000 INFO 4486 --- [prometheus-minute-schedule-1,e4ddcbc479045795,76dbb8f374486156] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:14.457 ERROR 4486 --- [metric-archive-second-21,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '669741696482349056-1729058280' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:59:14.456369] [YB420AC531B8-00060B7DBC38735C-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '669741696482349056-1729058280' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 13:59:14.456369] [YB420AC531B8-00060B7DBC38735C-0-0] 2024-10-16 13:59:14.475 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058280, nodeCount=1000, affectRows=1000 2024-10-16 13:59:14.479 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058280, nodeCount=1000, validCount=1000 2024-10-16 13:59:14.483 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058280, nodeCount=1000, affectRows=1000 2024-10-16 13:59:14.486 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058280, nodeCount=1000, affectRows=1000 2024-10-16 13:59:14.488 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058280, nodeCount=998, validCount=998 2024-10-16 13:59:14.491 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058280, nodeCount=1000, validCount=1000 2024-10-16 13:59:14.704 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058280, nodeCount=452, affectRows=452 2024-10-16 13:59:14.706 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058280, nodeCount=451, validCount=451 2024-10-16 13:59:14.934 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 13:59:14.934 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 13:59:14.941 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:59:14.942 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:59:14.942 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:59:14.942 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:59:14.947 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 13:59:14.947 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 13:59:14.947 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 13:59:14.948 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 13:59:14.995 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:59:15.003 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:59:15.003 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:59:15.003 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:59:15.003 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:59:15.003 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:59:15.010 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:59:15.195 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058280, nodeCount=480, affectRows=480 2024-10-16 13:59:15.366 INFO 4486 --- [ocp-async-4,600eae3850f3585d,bfdccd6c619c895c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:15.457 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 13:59:15.894 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058280, nodeCount=480, validCount=480 2024-10-16 13:59:16.033 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.085478402S 2024-10-16 13:59:16.033 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 13:59:16.104 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 13:59:16.104 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 13:59:16.128 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 13:59:16.194 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 13:59:16.194 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 13:59:16.194 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 13:59:16.194 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 13:59:16.194 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 13:59:16.194 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 13:59:16.202 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 13:59:16.321 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] o.s.boot.SpringApplication : Started application in 1.38 seconds (JVM running for 79259.373) 2024-10-16 13:59:16.322 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.380505607S 2024-10-16 13:59:16.443 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:16.648 INFO 4486 --- [ocp-schedules-20,ff554e0a328e99ac,b3cff2261e1413ab] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 13:59:16.994 INFO 4486 --- [prometheus-minute-schedule-1,741e2e83fd4b6f22,6146956dce406454] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:16.999 INFO 4486 --- [prometheus-minute-schedule-1,741e2e83fd4b6f22,6146956dce406454] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:17.069 INFO 4486 --- [ocp-schedules-17,af18210788e24512,dca9c8d834049f42] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:59:17.166 INFO 4486 --- [ocp-schedules-30,10586fc94267d411,452626b2ef9b0f60] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:59:17.265 INFO 4486 --- [ocp-schedules-15,dc420d9e6703fca5,d14ed9a63ce3e342] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:59:17.366 INFO 4486 --- [ocp-async-1,eeb2857a44571e14,485df4b5684f1dde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:17.521 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,957d0de61fbb0ddc,7efaa6db99a72661] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=957d0de61fbb0ddc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:59:17.522 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,957d0de61fbb0ddc,7efaa6db99a72661] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=957d0de61fbb0ddc, duration=3 ms] 2024-10-16 13:59:18.301 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,84015adc007a3e89,235242f60ad8228b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=84015adc007a3e89, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:18.301 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,84015adc007a3e89,235242f60ad8228b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=84015adc007a3e89, duration=1 ms] 2024-10-16 13:59:18.973 INFO 4486 --- [ocp-schedules-21,60cc2e787ab5c09b,9f5cffbefda67c89] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 13:59:18.975 INFO 4486 --- [ocp-schedules-21,60cc2e787ab5c09b,9f5cffbefda67c89] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:59:19.205 INFO 4486 --- [ocp-schedules-22,ce4c110aac3a2069,6a2475e7fc3ef516] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:59:19.229 INFO 4486 --- [ocp-schedules-8,102f1d488415c081,1992237a97d596b2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:59:19.234 WARN 4486 --- [scanner-13315147-1,ce4c110aac3a2069,8abfd0b4cd9d06ee] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:59:19.305 INFO 4486 --- [alarm-detect-3,ce4c110aac3a2069,e5580bf63c48e3ff] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058325 ~ 1729058349 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058340, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058340, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058340=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:59:19.366 INFO 4486 --- [ocp-async-2,7418278dc7ecad71,c6d17b88895bb1f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:19.444 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:19.994 INFO 4486 --- [prometheus-minute-schedule-1,29879d68381eb177,cc4e7b58e88e9b22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:20.002 INFO 4486 --- [prometheus-minute-schedule-1,29879d68381eb177,cc4e7b58e88e9b22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:20.321 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dd2b360207f2c0c2,add643973f4e645f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dd2b360207f2c0c2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:20.321 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,dd2b360207f2c0c2,add643973f4e645f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dd2b360207f2c0c2, duration=2 ms] 2024-10-16 13:59:20.996 INFO 4486 --- [prometheus-cache-monitor1,284b3cc2120978eb,b186e4ee5875cad5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13858,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8468} 2024-10-16 13:59:20.996 INFO 4486 --- [prometheus-cache-monitor1,284b3cc2120978eb,b186e4ee5875cad5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:59:21.199 INFO 4486 --- [ocp-schedules-14,0e4685d86fe5f2fb,578bc7c85b4037e5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:59:21.367 INFO 4486 --- [ocp-async-8,513886e4be6184db,3baaf4940c264425] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:21.859 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3aaf48774a90ddca,563c705aebc4a509] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3aaf48774a90ddca, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:21.860 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3aaf48774a90ddca,563c705aebc4a509] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3aaf48774a90ddca, duration=2 ms] 2024-10-16 13:59:22.174 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8981ec2bd8581dcd,e87649f1e0a34705] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8981ec2bd8581dcd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:22.174 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8981ec2bd8581dcd,e87649f1e0a34705] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8981ec2bd8581dcd, duration=1 ms] 2024-10-16 13:59:22.444 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:22.994 INFO 4486 --- [prometheus-minute-schedule-1,f8c76a55cdd98c16,cc62067136831d8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:23.000 INFO 4486 --- [prometheus-minute-schedule-1,f8c76a55cdd98c16,cc62067136831d8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:23.025 INFO 4486 --- [ocp-async-5,be6595daae5725e3,9993ec857b476078] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:59:23.367 INFO 4486 --- [ocp-async-3,4e0d3b6ee592df28,8370432592203650] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:23.858 INFO 4486 --- [ocp-schedules-22,461b33faab64d0f0,3bf572c32f4f5c71] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 13:59:25.147 INFO 4486 --- [ocp-schedules-27,082320b32a23109f,c3d406432a56e666] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 13:59:25.367 INFO 4486 --- [ocp-async-10,45dc77131a2d8c66,01c12c077c906dba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:25.445 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:25.994 INFO 4486 --- [prometheus-minute-schedule-1,4b8a6ed14e4cd05a,20ddb99c2619e86b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:26.000 INFO 4486 --- [prometheus-minute-schedule-1,4b8a6ed14e4cd05a,20ddb99c2619e86b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:26.613 INFO 4486 --- [ocp-schedules-23,f932807349cd0b8c,c3cb8efef86e71a7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 13:59:26.613 INFO 4486 --- [ocp-schedules-23,f932807349cd0b8c,c3cb8efef86e71a7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 13:59:26.646 INFO 4486 --- [Collect all tenants major compactions1,f932807349cd0b8c,1088c1b4ae7ab4fc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:59:26.646 INFO 4486 --- [Collect all tenants major compactions2,f932807349cd0b8c,36e64b3ff1f228d5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:59:26.646 INFO 4486 --- [Collect all tenants major compactions4,f932807349cd0b8c,1887007b28ba109d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:59:26.647 INFO 4486 --- [Collect all tenants major compactions3,f932807349cd0b8c,e4637d3a85791b83] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:59:26.647 INFO 4486 --- [Collect all tenants major compactions5,f932807349cd0b8c,e405056497fa5fe7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:59:26.666 INFO 4486 --- [Collect all tenants major compactions1,f932807349cd0b8c,1088c1b4ae7ab4fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:26.668 INFO 4486 --- [Collect all tenants major compactions1,f932807349cd0b8c,1088c1b4ae7ab4fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:59:26.668 INFO 4486 --- [Collect all tenants major compactions2,f932807349cd0b8c,36e64b3ff1f228d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:26.670 INFO 4486 --- [Collect all tenants major compactions2,f932807349cd0b8c,36e64b3ff1f228d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:59:26.670 INFO 4486 --- [Collect all tenants major compactions3,f932807349cd0b8c,e4637d3a85791b83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:26.671 INFO 4486 --- [Collect all tenants major compactions3,f932807349cd0b8c,e4637d3a85791b83] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 13:59:26.672 INFO 4486 --- [Collect all tenants major compactions1,f932807349cd0b8c,1088c1b4ae7ab4fc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 13:59:26.672 INFO 4486 --- [Collect all tenants major compactions2,f932807349cd0b8c,36e64b3ff1f228d5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 13:59:26.675 INFO 4486 --- [Collect all tenants major compactions3,f932807349cd0b8c,e4637d3a85791b83] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:59:26.679 INFO 4486 --- [Collect all tenants major compactions4,f932807349cd0b8c,1887007b28ba109d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:26.679 INFO 4486 --- [Collect all tenants major compactions5,f932807349cd0b8c,e405056497fa5fe7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:26.680 INFO 4486 --- [Collect all tenants major compactions1,f932807349cd0b8c,1088c1b4ae7ab4fc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 13:59:26.681 INFO 4486 --- [Collect all tenants major compactions4,f932807349cd0b8c,1887007b28ba109d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 13:59:26.681 INFO 4486 --- [Collect all tenants major compactions5,f932807349cd0b8c,e405056497fa5fe7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 13:59:26.681 INFO 4486 --- [Collect all tenants major compactions3,f932807349cd0b8c,e4637d3a85791b83] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 13:59:26.683 INFO 4486 --- [Collect all tenants major compactions2,f932807349cd0b8c,36e64b3ff1f228d5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 13:59:26.684 INFO 4486 --- [Collect all tenants major compactions4,f932807349cd0b8c,1887007b28ba109d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 13:59:26.684 INFO 4486 --- [Collect all tenants major compactions4,f932807349cd0b8c,1887007b28ba109d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 13:59:26.685 INFO 4486 --- [Collect all tenants major compactions5,f932807349cd0b8c,e405056497fa5fe7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 13:59:26.686 INFO 4486 --- [Collect all tenants major compactions5,f932807349cd0b8c,e405056497fa5fe7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 13:59:26.686 INFO 4486 --- [ocp-schedules-23,f932807349cd0b8c,c3cb8efef86e71a7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 13:59:27.368 INFO 4486 --- [ocp-async-6,da3aa46631cd25df,13ffcd887e59f14b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:27.796 INFO 4486 --- [ocp-schedules-12,3786cc356aa462fe,48c3306066e58f27] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T13:59:27.796+08:00, clusters=[1, 5, 1000003] 2024-10-16 13:59:27.799 INFO 4486 --- [ocp-schedules-12,3786cc356aa462fe,48c3306066e58f27] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 13:59:27.800 INFO 4486 --- [ocp-schedules-12,3786cc356aa462fe,48c3306066e58f27] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 13:59:27.800 INFO 4486 --- [ocp-schedules-12,3786cc356aa462fe,48c3306066e58f27] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 13:59:28.307 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3b59278aa0a8d283,5cc411a5ea883d9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3b59278aa0a8d283, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:28.308 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3b59278aa0a8d283,5cc411a5ea883d9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3b59278aa0a8d283, duration=2 ms] 2024-10-16 13:59:28.446 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:28.765 INFO 4486 --- [ocp-schedules-11,4b153fa4119c0be8,694d2dee5029aaaf] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 13:59:28.994 INFO 4486 --- [prometheus-minute-schedule-1,f0abdeddb3d8714f,cc09506973e97818] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:28.999 INFO 4486 --- [prometheus-minute-schedule-1,f0abdeddb3d8714f,cc09506973e97818] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:29.008 INFO 4486 --- [ocp-schedules-21,60cc2e787ab5c09b,9f5cffbefda67c89] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 13:59:29.368 INFO 4486 --- [ocp-async-9,64eba082d6dc58b7,23df7f2bfff36b70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:29.379 INFO 4486 --- [ocp-schedules-13,00635db271c90034,8b4c77174c4aa822] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:59:29.404 WARN 4486 --- [scanner-13315447-1,00635db271c90034,af2a3afc409e40ad] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:59:29.470 INFO 4486 --- [alarm-detect-15,00635db271c90034,717c03b0be3f7df6] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058340 ~ 1729058359 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058355, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058355, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058355=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:59:30.102 INFO 4486 --- [prometheus-second-schedule-1,775739a04e09d648,1a9aecdb3fe564d0] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 13:59:30.327 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,81b414a1e456db11,c21d72990110f670] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=81b414a1e456db11, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:30.327 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,81b414a1e456db11,c21d72990110f670] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=81b414a1e456db11, duration=1 ms] 2024-10-16 13:59:31.204 INFO 4486 --- [ocp-schedules-19,a17f22ef36fe3695,85356cdb2dd6af31] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:59:31.368 INFO 4486 --- [ocp-async-3,1ef01358a941ff62,8c87b306be42e58f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:31.447 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:31.865 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,066a23ef1f087e33,7d4269ba77f103dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=066a23ef1f087e33, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:31.865 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,066a23ef1f087e33,7d4269ba77f103dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=066a23ef1f087e33, duration=2 ms] 2024-10-16 13:59:31.994 INFO 4486 --- [prometheus-minute-schedule-1,9b3b0a3b0c0de386,9233b9290073f17a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:32.001 INFO 4486 --- [prometheus-minute-schedule-1,9b3b0a3b0c0de386,9233b9290073f17a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:32.179 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b1610a89cb5d43c5,55ede91f74c480bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b1610a89cb5d43c5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:32.179 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b1610a89cb5d43c5,55ede91f74c480bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b1610a89cb5d43c5, duration=1 ms] 2024-10-16 13:59:33.025 INFO 4486 --- [ocp-async-2,0f14eae9486ffcce,18c03570fd4c585a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:59:33.368 INFO 4486 --- [ocp-async-8,3356908ba2a2d51f,8225461c3eb0470c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:34.448 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:34.994 INFO 4486 --- [prometheus-minute-schedule-1,8051d0ca2affb16c,f493d104c41b4665] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:35.001 INFO 4486 --- [prometheus-minute-schedule-1,8051d0ca2affb16c,f493d104c41b4665] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:35.368 INFO 4486 --- [ocp-async-1,87b3afe9b5549b69,0c683d27efe4fc93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:36.796 INFO 4486 --- [ocp-schedules-16,5896ab4dfbbd0459,1086fcae0c8f8c3f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:59:36.796 INFO 4486 --- [ocp-schedules-16,deb8f939ced5587c,95f49a1cbcec689a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 13:59:36.798 INFO 4486 --- [ocp-schedules-16,deb8f939ced5587c,95f49a1cbcec689a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:59:36.835 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:36.836 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:36.842 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:36.843 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:36.871 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:36.872 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:36.874 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:36.882 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:36.883 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:59:36.921 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:36.923 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:36.924 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:36.925 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 13:59:36.929 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:36.932 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:59:36.933 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:36.949 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:36.950 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:36.966 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:36.975 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:36.976 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:36.976 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:36.977 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 13:59:36.982 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 13:59:36.986 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:36.987 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:37.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.003 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.004 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 13:59:37.015 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:59:37.046 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 13:59:37.046 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:59:37.051 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.053 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.054 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 13:59:37.058 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.062 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 13:59:37.063 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 13:59:37.064 INFO 4486 --- [ocp-schedules-19,9b39c3ca89eb5e73,5a955903936d15e2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 13:59:37.065 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:37.066 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 13:59:37.067 INFO 4486 --- [ocp-schedules-3,3b2baf7fa35c06c5,7daf8b8930fcdef3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 13:59:37.069 INFO 4486 --- [ocp-schedules-3,3b2baf7fa35c06c5,7daf8b8930fcdef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:59:37.070 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 13:59:37.072 INFO 4486 --- [ocp-schedules-19,9b39c3ca89eb5e73,5a955903936d15e2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 13:59:37.081 INFO 4486 --- [ocp-schedules-8,df90475949b4947d,76265d4b4e510020] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 13:59:37.083 INFO 4486 --- [ocp-schedules-8,df90475949b4947d,76265d4b4e510020] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 13:59:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:59:37.091 INFO 4486 --- [ocp-schedules-2,9db278d816fbd6f3,e6a70a9f15a2040d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:59:37.091 INFO 4486 --- [ocp-schedules-18,940c805b1575b122,18fedccb8bd925a0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:59:37.091 INFO 4486 --- [ocp-schedules-25,9f50b3d5d32b2f33,e1ea2545a6907aaa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:59:37.092 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:59:37.092 INFO 4486 --- [host-check-clockdiff1,9db278d816fbd6f3,a8c7e261270ea223] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 13:59:37.092 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:59:37.092 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:59:37.094 INFO 4486 --- [host-check-agent-status1,940c805b1575b122,a70b4fbfae0c84f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 13:59:37.094 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:59:37.095 INFO 4486 --- [host-check-agent-status3,940c805b1575b122,29bc4aa2f63ece2a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 13:59:37.096 INFO 4486 --- [host-check-agent-status2,940c805b1575b122,cd2a1158255654b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 13:59:37.097 INFO 4486 --- [host-check-agent-status4,940c805b1575b122,ebb95aa8a79d43a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 13:59:37.097 INFO 4486 --- [host-check-clockdiff2,9db278d816fbd6f3,fe4b8404f727d98b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 13:59:37.098 INFO 4486 --- [host-check-clockdiff3,9db278d816fbd6f3,651865ba7cff0c30] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 13:59:37.098 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:37.098 INFO 4486 --- [host-check-clockdiff4,9db278d816fbd6f3,57ef527b248436ed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 13:59:37.099 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:59:37.100 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:37.101 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:59:37.102 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:59:37.102 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:37.102 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:37.103 INFO 4486 --- [host-check-agent-status4,940c805b1575b122,ebb95aa8a79d43a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:59:37.104 INFO 4486 --- [host-check-agent-status3,940c805b1575b122,29bc4aa2f63ece2a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:59:37.104 INFO 4486 --- [host-check-agent-status1,940c805b1575b122,a70b4fbfae0c84f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:37.104 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:59:37.105 INFO 4486 --- [host-check-agent-status4,940c805b1575b122,ebb95aa8a79d43a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:59:37.106 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:59:37.106 INFO 4486 --- [host-check-agent-status2,940c805b1575b122,cd2a1158255654b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:59:37.115 INFO 4486 --- [host-check-agent-status3,940c805b1575b122,29bc4aa2f63ece2a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:59:37.115 INFO 4486 --- [host-check-agent-status2,940c805b1575b122,cd2a1158255654b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:59:37.115 INFO 4486 --- [host-check-agent-status1,940c805b1575b122,a70b4fbfae0c84f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:37.116 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 13:59:37.116 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:59:37.135 INFO 4486 --- [host-check-clockdiff1,9db278d816fbd6f3,a8c7e261270ea223] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058377 0 0 , error= 2024-10-16 13:59:37.145 INFO 4486 --- [host-check-clockdiff4,9db278d816fbd6f3,57ef527b248436ed] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058377 4 4 , error= 2024-10-16 13:59:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:37.154 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 13:59:37.161 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.165 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 13:59:37.166 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.167 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:59:37.168 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 13:59:37.168 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T13:59:37.172560249+08:00 2024-10-16 13:59:37.169 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T13:59:37.169+08:00 2024-10-16 13:59:37.170 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T13:59:37.169886692+08:00 2024-10-16 13:59:37.170 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T13:59:37.170+08:00 2024-10-16 13:59:37.171 INFO 4486 --- [host-check-agent-status3,940c805b1575b122,29bc4aa2f63ece2a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 13:59:37.173 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 13:59:37.173 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 13:59:37.174 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.175 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T13:59:37.179199542+08:00 2024-10-16 13:59:37.175 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:37.175 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T13:59:37.179171968+08:00 2024-10-16 13:59:37.175 INFO 4486 --- [host-check-agent-status3,940c805b1575b122,29bc4aa2f63ece2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 13:59:37.176 INFO 4486 --- [host-check-agent-status3,940c805b1575b122,29bc4aa2f63ece2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 13:59:37.176 INFO 4486 --- [host-check-agent-status3,940c805b1575b122,29bc4aa2f63ece2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 13:59:37.176 INFO 4486 --- [host-check-agent-status4,940c805b1575b122,ebb95aa8a79d43a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 13:59:37.176 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T13:59:37.175+08:00 2024-10-16 13:59:37.176 INFO 4486 --- [host-check-agent-status2,940c805b1575b122,cd2a1158255654b7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 13:59:37.177 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T13:59:37.175+08:00 2024-10-16 13:59:37.177 INFO 4486 --- [host-check-agent-status1,940c805b1575b122,a70b4fbfae0c84f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 13:59:37.178 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T13:59:37.169+08:00 and status AVAILABLE 2024-10-16 13:59:37.178 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 13:59:37.178 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 13:59:37.179 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T13:59:37.175+08:00 and status AVAILABLE 2024-10-16 13:59:37.179 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 13:59:37.179 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T13:59:37.170+08:00 and status AVAILABLE 2024-10-16 13:59:37.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:37.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:37.180 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T13:59:37.175+08:00 and status AVAILABLE 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-agent-status4,940c805b1575b122,ebb95aa8a79d43a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-agent-status4,940c805b1575b122,ebb95aa8a79d43a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-agent-status4,940c805b1575b122,ebb95aa8a79d43a9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-agent-status2,940c805b1575b122,cd2a1158255654b7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-agent-status2,940c805b1575b122,cd2a1158255654b7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-agent-status2,940c805b1575b122,cd2a1158255654b7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 13:59:37.181 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 13:59:37.182 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 13:59:37.182 INFO 4486 --- [host-check-agent-status1,940c805b1575b122,a70b4fbfae0c84f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 13:59:37.182 INFO 4486 --- [host-check-agent-status1,940c805b1575b122,a70b4fbfae0c84f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 13:59:37.182 INFO 4486 --- [host-check-agent-status1,940c805b1575b122,a70b4fbfae0c84f9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 13:59:37.183 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 13:59:37.189 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.190 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:59:37.190 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.190 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:37.191 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:59:37.191 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.192 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.192 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:37.192 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:59:37.192 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:59:37.193 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:59:37.193 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:59:37.194 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:59:37.195 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:59:37.199 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 13:59:37.203 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.203 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.203 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.203 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.203 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.203 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.212 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:37.212 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:37.212 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.213 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:59:37.214 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:37.216 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:37.217 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:37.218 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 13:59:37.220 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 13:59:37.228 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.229 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:59:37.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:37.243 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 13:59:37.244 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 13:59:37.245 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.246 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 13:59:37.248 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.250 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 13:59:37.255 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 13:59:37.258 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 13:59:37.263 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.264 INFO 4486 --- [ocp-schedules-21,4f42e1a16a399cd5,78d6c21b1fcc0909] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 13:59:37.265 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 13:59:37.269 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 13:59:37.273 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:37.273 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 13:59:37.274 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 13:59:37.279 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 13:59:37.281 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:37.286 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 13:59:37.288 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:37.296 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:37.302 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 13:59:37.304 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 13:59:37.315 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 13:59:37.322 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 13:59:37.327 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 13:59:37.330 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.332 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.332 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 13:59:37.333 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 13:59:37.333 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 13:59:37.337 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 13:59:37.342 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 13:59:37.343 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:37.343 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 13:59:37.346 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 13:59:37.348 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 13:59:37.350 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:37.360 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:37.366 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 13:59:37.369 INFO 4486 --- [ocp-async-3,235f73961c1180d7,0ee18f7373493a8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:37.369 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 13:59:37.372 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 13:59:37.380 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 13:59:37.381 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 13:59:37.383 INFO 4486 --- [host-check-host-status3,9f50b3d5d32b2f33,bc36ee98c88e5740] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 13:59:37.391 INFO 4486 --- [host-check-host-status4,9f50b3d5d32b2f33,d57f285bec0a5d81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 13:59:37.396 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 13:59:37.403 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 13:59:37.414 INFO 4486 --- [host-check-host-status2,9f50b3d5d32b2f33,2a530302d1780766] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 13:59:37.445 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.448 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.449 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:59:37.449 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:37.449 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 13:59:37.450 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.451 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.452 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:59:37.460 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.460 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.460 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.460 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.460 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.460 INFO 4486 --- [host-check-host-status1,9f50b3d5d32b2f33,4b2472b50bb4bbe2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 13:59:37.460 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.486 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 13:59:37.513 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 13:59:37.532 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 13:59:37.541 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.543 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.544 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 13:59:37.571 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 13:59:37.572 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 13:59:37.576 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.591 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.593 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.594 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 13:59:37.595 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 13:59:37.600 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.601 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:59:37.602 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.603 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.604 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:59:37.613 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.613 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.613 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.613 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.613 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.614 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.621 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 13:59:37.819 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 13:59:37.867 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:37.869 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.870 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:59:37.872 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.872 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.873 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:59:37.882 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.882 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.882 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.882 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.882 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:37.882 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:37.899 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 13:59:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:59:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 13:59:37.994 INFO 4486 --- [prometheus-minute-schedule-1,292895fd6c86dc06,75db8f9df8951ddc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:38.036 INFO 4486 --- [prometheus-minute-schedule-1,292895fd6c86dc06,75db8f9df8951ddc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:38.058 INFO 4486 --- [host-check-clockdiff2,9db278d816fbd6f3,fe4b8404f727d98b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058378 4 4 , error= 2024-10-16 13:59:38.074 INFO 4486 --- [host-check-clockdiff3,9db278d816fbd6f3,651865ba7cff0c30] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058378 4 4 , error= 2024-10-16 13:59:38.169 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 13:59:38.220 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 13:59:38.222 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:38.223 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 13:59:38.224 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:38.225 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:38.226 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 13:59:38.234 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:38.234 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:38.234 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:38.234 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:38.234 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 13:59:38.235 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:38.297 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 13:59:38.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ae16646248a23543,05256173708d752b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ae16646248a23543, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:38.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ae16646248a23543,05256173708d752b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ae16646248a23543, duration=2 ms] 2024-10-16 13:59:38.459 INFO 4486 --- [ocp-schedules-21,ab5ecdf97f6de375,49195aab8b891f34] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 13:59:38.459 INFO 4486 --- [ocp-schedules-21,ab5ecdf97f6de375,49195aab8b891f34] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 13:59:38.460 INFO 4486 --- [ocp-schedules-21,ab5ecdf97f6de375,49195aab8b891f34] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 13:59:38.479 INFO 4486 --- [ocp-schedules-11,a0be356f31825cb4,e69a9abdb396ff4c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13858,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8468} 2024-10-16 13:59:38.479 INFO 4486 --- [ocp-schedules-11,a0be356f31825cb4,e69a9abdb396ff4c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 13:59:38.497 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 13:59:38.301888, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 13:59:38.497 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 13:59:38.301888, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 13:59:38.534 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 13:59:38.301888} } 2024-10-16 13:59:38.593 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 13:59:38.601 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:59:38.604 INFO 4486 --- [ocp-schedules-13,0a6b5535c2e1bb60,2bfe8e6ecfd94389] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 13:59:39.235 INFO 4486 --- [ocp-schedules-15,5a35855cb3d0bfb1,d239fb07cb2d8a10] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:59:39.369 INFO 4486 --- [ocp-async-8,9a484a44752472f8,b84500c10b3ad1af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:39.581 INFO 4486 --- [ocp-schedules-23,6e9e4858562f0539,ef3ce026d3ab16be] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:59:39.619 WARN 4486 --- [scanner-13315713-1,6e9e4858562f0539,dbf85c6ce7be55d6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:59:40.006 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 13:59:40.006 INFO 4486 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 13:59:40.024 INFO 4486 --- [Check obproxy1,ecb678095af6c593,d40f38806f8bafb2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:59:40.025 INFO 4486 --- [Check obproxy2,81d844ab53bd57ba,3436a5319a5153c9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:59:40.027 INFO 4486 --- [Collect obproxy1,cd6dafb779beb0c3,238fae38b0e25bd1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 13:59:40.029 INFO 4486 --- [Check obproxy1,ecb678095af6c593,d40f38806f8bafb2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:40.029 INFO 4486 --- [Collect obproxy2,4f292b28059c008c,194300bf5904bfe6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 13:59:40.029 INFO 4486 --- [Collect obproxy2,4f292b28059c008c,194300bf5904bfe6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:40.032 INFO 4486 --- [Check obproxy2,81d844ab53bd57ba,3436a5319a5153c9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:40.033 INFO 4486 --- [Collect obproxy1,cd6dafb779beb0c3,238fae38b0e25bd1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 13:59:40.053 INFO 4486 --- [Collect obproxy2,4f292b28059c008c,194300bf5904bfe6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:59:40.054 INFO 4486 --- [Check obproxy2,81d844ab53bd57ba,3436a5319a5153c9] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 13:59:40.054 INFO 4486 --- [Check obproxy2,81d844ab53bd57ba,3436a5319a5153c9] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 13:59:40.059 INFO 4486 --- [Collect obproxy2,4f292b28059c008c,194300bf5904bfe6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:59:40.059 INFO 4486 --- [Collect obproxy1,cd6dafb779beb0c3,238fae38b0e25bd1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 13:59:40.062 INFO 4486 --- [Collect obproxy1,cd6dafb779beb0c3,238fae38b0e25bd1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 13:59:40.064 INFO 4486 --- [Check obproxy1,ecb678095af6c593,d40f38806f8bafb2] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 13:59:40.064 INFO 4486 --- [Check obproxy1,ecb678095af6c593,d40f38806f8bafb2] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 13:59:40.064 INFO 4486 --- [Collect obproxy2,4f292b28059c008c,194300bf5904bfe6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:59:40.065 INFO 4486 --- [Collect obproxy1,cd6dafb779beb0c3,238fae38b0e25bd1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 13:59:40.331 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e9e01c3cbd2bc404,fed2bb132aab9680] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e9e01c3cbd2bc404, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:40.332 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,e9e01c3cbd2bc404,fed2bb132aab9680] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e9e01c3cbd2bc404, duration=1 ms] 2024-10-16 13:59:40.450 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:40.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a6a9e427862e4381,8f2a68e987f5fd16] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=a6a9e427862e4381, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:59:40.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a6a9e427862e4381,8f2a68e987f5fd16] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=a6a9e427862e4381, duration=1 ms] 2024-10-16 13:59:40.994 INFO 4486 --- [prometheus-minute-schedule-1,3eef275e204ffde4,17fd609bdbd894dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:40.995 INFO 4486 --- [prometheus-hour-schedule-1,8047d9efc7a2caef,130b2fd4b393cb5b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=3600 2024-10-16 13:59:40.999 INFO 4486 --- [prometheus-minute-schedule-1,3eef275e204ffde4,17fd609bdbd894dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:41.209 INFO 4486 --- [ocp-schedules-15,e65d3f36b87c8a32,1ac70b94a945455f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:59:41.369 INFO 4486 --- [ocp-async-9,1188cc55e41037e9,4f773b44f5dfb289] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:41.870 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ccf46cd72ff9ca34,2638b988d4d8270e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ccf46cd72ff9ca34, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:41.870 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ccf46cd72ff9ca34,2638b988d4d8270e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ccf46cd72ff9ca34, duration=1 ms] 2024-10-16 13:59:42.179 INFO 4486 --- [ocp-schedules-24,b52161149ac5d2cd,c95b4ba4fabf65c8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 13:59:42.183 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,67eb9b32dc629c45,a8c7197c220f2a45] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=67eb9b32dc629c45, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:42.183 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,67eb9b32dc629c45,a8c7197c220f2a45] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=67eb9b32dc629c45, duration=1 ms] 2024-10-16 13:59:43.025 INFO 4486 --- [ocp-async-4,97d6de0652c43efd,bcd0ad5f337700cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:59:43.369 INFO 4486 --- [ocp-async-3,87da2c87cc79f550,1a8288db6ce872d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:43.450 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:43.994 INFO 4486 --- [prometheus-minute-schedule-1,a0d32ec95bfdad04,b0be36a57ab841ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:44.000 INFO 4486 --- [prometheus-minute-schedule-1,a0d32ec95bfdad04,b0be36a57ab841ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:45.369 INFO 4486 --- [ocp-async-6,dedd4fee0882b79a,98e759fe737f37f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:45.584 INFO 4486 --- [ocp-schedules-27,0bec6caa7e1db00e,6ebbf22683c3ad24] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 13:59:45.589 INFO 4486 --- [ocp-schedules-27,0bec6caa7e1db00e,6ebbf22683c3ad24] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 13:59:45.589 INFO 4486 --- [ocp-schedules-27,0bec6caa7e1db00e,6ebbf22683c3ad24] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 13:59:45.615 INFO 4486 --- [ocp-schedules-17,364bd6f573332719,f9e6225198599d23] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 13:59:45.615 INFO 4486 --- [ocp-schedules-17,364bd6f573332719,f9e6225198599d23] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 13:59:46.451 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:46.994 INFO 4486 --- [prometheus-minute-schedule-1,5782cac0bb7734a7,dfde139713d00580] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:46.999 INFO 4486 --- [prometheus-minute-schedule-1,5782cac0bb7734a7,dfde139713d00580] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:47.071 INFO 4486 --- [ocp-schedules-3,3b2baf7fa35c06c5,7daf8b8930fcdef3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 13:59:47.180 INFO 4486 --- [ocp-schedules-8,df90475949b4947d,76265d4b4e510020] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 13:59:47.268 INFO 4486 --- [ocp-schedules-12,1173489fd5031bf7,e5bc2149d678d1f2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 13:59:47.370 INFO 4486 --- [ocp-async-1,3f83cfbff5d65de9,6ff4efa772b0f565] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:48.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,276c0632fb92e4a4,de2cfbd46ae22244] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=276c0632fb92e4a4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:48.319 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,276c0632fb92e4a4,de2cfbd46ae22244] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=276c0632fb92e4a4, duration=2 ms] 2024-10-16 13:59:48.783 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:48.783 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:59:48.785 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 13:59:48.786 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 13:59:48.786 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 13:59:49.370 INFO 4486 --- [ocp-async-7,30ae308ecb7afe2a,85f2f3755edb1a7c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:49.452 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:49.749 INFO 4486 --- [ocp-schedules-11,52c33a38bb2a24af,520148cd00a7876a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:59:49.792 WARN 4486 --- [scanner-13316019-1,52c33a38bb2a24af,08e69c76e9a5db5f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 13:59:49.858 INFO 4486 --- [alarm-detect-8,52c33a38bb2a24af,52703170501d2cea] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058355 ~ 1729058379 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058370, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058370, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058370=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 13:59:49.994 INFO 4486 --- [prometheus-minute-schedule-1,621f38d9e5c067df,cc5b2f3759e4feda] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:50.000 INFO 4486 --- [prometheus-minute-schedule-1,621f38d9e5c067df,cc5b2f3759e4feda] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:50.337 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bd47dc450f189b85,17b3bab2d2cd50a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bd47dc450f189b85, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:50.337 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,bd47dc450f189b85,17b3bab2d2cd50a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bd47dc450f189b85, duration=1 ms] 2024-10-16 13:59:50.550 INFO 4486 --- [ocp-schedules-22,a9b4864911773af1,ebad21004235a76f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 13:59:50.550 INFO 4486 --- [ocp-schedules-22,a9b4864911773af1,ebad21004235a76f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 13:59:50.555 INFO 4486 --- [ocp-schedules-22,a9b4864911773af1,ebad21004235a76f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 13:59:50.555 INFO 4486 --- [ocp-schedules-22,a9b4864911773af1,ebad21004235a76f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 13:59:50.559 INFO 4486 --- [ocp-schedules-22,a9b4864911773af1,ebad21004235a76f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 13:59:50.559 INFO 4486 --- [ocp-schedules-22,a9b4864911773af1,ebad21004235a76f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 13:59:51.214 INFO 4486 --- [ocp-schedules-15,168c73a618e8068e,78e7131c6a9d3983] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 13:59:51.370 INFO 4486 --- [ocp-async-5,81c8a9f6b81881e2,9584d573c7934309] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:51.388 INFO 4486 --- [ocp-schedules-25,2a662f4c3a183d99,0e1e276dfbb95ca5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 13:59:51.388 INFO 4486 --- [ocp-schedules-25,2a662f4c3a183d99,0e1e276dfbb95ca5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 13:59:51.388 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 13:59:51.388 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 13:59:51.388 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 13:59:51.388 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 13:59:51.388 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 13:59:51.393 INFO 4486 --- [ocp-schedules-25,2a662f4c3a183d99,0e1e276dfbb95ca5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 13:59:51.394 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 13:59:51.394 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 13:59:51.395 INFO 4486 --- [ocp-schedules-25,2a662f4c3a183d99,0e1e276dfbb95ca5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 13:59:51.395 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 13:59:51.395 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 13:59:51.395 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 13:59:51.397 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 13:59:51.402 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 13:59:51.403 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 13:59:51.404 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 13:59:51.404 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 13:59:51.420 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 13:59:51.420 INFO 4486 --- [ocp-schedules-25,2a662f4c3a183d99,0e1e276dfbb95ca5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 13:59:51.420 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 13:59:51.422 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 13:59:51.428 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 13:59:51.428 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 13:59:51.430 INFO 4486 --- [ocp-schedules-25,2a662f4c3a183d99,0e1e276dfbb95ca5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 13:59:51.874 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,5ae3e09a9612a3b4,792f8cd9e637413e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5ae3e09a9612a3b4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:51.874 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,5ae3e09a9612a3b4,792f8cd9e637413e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5ae3e09a9612a3b4, duration=1 ms] 2024-10-16 13:59:52.187 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,78615f5daa77c4c3,f930d27773dac9be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=78615f5daa77c4c3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:52.187 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,78615f5daa77c4c3,f930d27773dac9be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=78615f5daa77c4c3, duration=1 ms] 2024-10-16 13:59:52.453 INFO 4486 --- [ocp-schedules-19,7b12e57f390bf91f,5f2826ad2c72ca0e] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 13:59:52.453 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:52.994 INFO 4486 --- [prometheus-minute-schedule-1,4e0051b19e113f4b,40b6cb2e20c0d35e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:53.000 INFO 4486 --- [prometheus-minute-schedule-1,4e0051b19e113f4b,40b6cb2e20c0d35e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:53.026 INFO 4486 --- [ocp-async-8,edbf96e864b2a29d,d7347fa733dce343] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 13:59:53.370 INFO 4486 --- [ocp-async-1,8a35af7d3410971f,6f8516cbb1699015] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:53.626 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,62a18dadb45e54c3,b7a6705f1d04a5a6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=62a18dadb45e54c3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 13:59:53.627 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,62a18dadb45e54c3,b7a6705f1d04a5a6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=62a18dadb45e54c3, duration=2 ms] 2024-10-16 13:59:55.371 INFO 4486 --- [ocp-async-10,e5919a1e346da96e,2469d444c4e0ae14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:55.454 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:55.994 INFO 4486 --- [prometheus-minute-schedule-1,13da5b6fc2475038,85efdfb5f194a8c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:56.000 INFO 4486 --- [prometheus-minute-schedule-1,13da5b6fc2475038,85efdfb5f194a8c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:57.371 INFO 4486 --- [ocp-async-8,2258c48a7dcbf018,5fc049f36e4b86e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:58.324 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1e828d5a2a3b68a7,f3efcda545797ae6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1e828d5a2a3b68a7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 13:59:58.325 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1e828d5a2a3b68a7,f3efcda545797ae6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1e828d5a2a3b68a7, duration=2 ms] 2024-10-16 13:59:58.456 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 13:59:58.994 INFO 4486 --- [prometheus-minute-schedule-1,b27a971ec151d19b,22999a36e0b07c07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058400, interval=60 2024-10-16 13:59:59.000 INFO 4486 --- [prometheus-minute-schedule-1,b27a971ec151d19b,22999a36e0b07c07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 13:59:59.240 INFO 4486 --- [ocp-schedules-14,e5aac6a473436dd4,1b4f234b5596ba13] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 13:59:59.371 INFO 4486 --- [ocp-async-9,383dace531a23b9e,c08ea14356428de3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 13:59:59.910 INFO 4486 --- [ocp-schedules-6,8d4ede3e93c5fac6,f2b17d6a66d50f95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 13:59:59.957 WARN 4486 --- [scanner-13316277-1,8d4ede3e93c5fac6,ebe77afa58b6894e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:00:00.037 INFO 4486 --- [alarm-detect-13,8d4ede3e93c5fac6,0fdf58c5837099d4] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058370 ~ 1729058390 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058385, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058385, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058385=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:00:00.342 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ab36483a8ace02ff,0ebb9e780d4078e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ab36483a8ace02ff, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:00.342 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ab36483a8ace02ff,0ebb9e780d4078e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ab36483a8ace02ff, duration=1 ms] 2024-10-16 14:00:01.220 INFO 4486 --- [ocp-schedules-9,0d14726cb7133914,557f889bba3bdb35] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:00:01.293 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e7b1ec6a0d5a5cc5,f0e9d568141e698b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=e7b1ec6a0d5a5cc5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:00:01.294 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e7b1ec6a0d5a5cc5,f0e9d568141e698b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=e7b1ec6a0d5a5cc5, duration=2 ms] 2024-10-16 14:00:01.371 INFO 4486 --- [ocp-async-2,d356d243b65e36a9,3f961b64690568ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:01.452 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,0e5b3c90409aef68] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934023, operation:EXECUTE, state: RUNNING 2024-10-16 14:00:01.457 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:01.468 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7934023, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266496, task_operation=execute, latest_execution_start_time=2024-10-16T14:00:01.445+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=7934023}, listMap={}}, executor=10.197.49.184 2024-10-16 14:00:01.501 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.575 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.731 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.779 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.790 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for OceanBase 2024-10-16 14:00:01.790 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.ocp.diagnosis.model.NeuralNetwork : W:[999.6945755151274], b:[-1.7128249860997754E12] 2024-10-16 14:00:01.794 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.878 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ba0a4d1f64c124aa,2b02edb24bd6e476] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ba0a4d1f64c124aa, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:01.878 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ba0a4d1f64c124aa,2b02edb24bd6e476] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ba0a4d1f64c124aa, duration=1 ms] 2024-10-16 14:00:01.910 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.939 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.964 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:00:01.994 INFO 4486 --- [prometheus-minute-schedule-1,003aadd3ff27d41b,d92ae0baf71f8475] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:02.000 INFO 4486 --- [prometheus-minute-schedule-1,003aadd3ff27d41b,d92ae0baf71f8475] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:02.027 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.064 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.191 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0e744d00ba65d235,e2272ffc30f39241] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0e744d00ba65d235, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:02.191 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0e744d00ba65d235,e2272ffc30f39241] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0e744d00ba65d235, duration=1 ms] 2024-10-16 14:00:02.313 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.344 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.386 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for obce_3zones 2024-10-16 14:00:02.386 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1437.0635791927157], b:[2.531038719703609E12] 2024-10-16 14:00:02.390 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.537 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.575 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.590 INFO 4486 --- [ocp-schedules-18,0c0335fd28d0e81c,c97c1f4ada72693e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:00:02.590+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:00:02.594 INFO 4486 --- [ocp-schedules-18,0c0335fd28d0e81c,c97c1f4ada72693e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:00:02.608 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:00:02.608 INFO 4486 --- [ocp-schedules-18,0c0335fd28d0e81c,c97c1f4ada72693e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:00:02.619 INFO 4486 --- [ocp-schedules-18,0c0335fd28d0e81c,c97c1f4ada72693e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:00:02.766 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:02.814 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:03.008 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:03.026 INFO 4486 --- [ocp-async-1,406e84af1723a60d,27b2634161d24510] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:00:03.042 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:03.070 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for glob_3zone 2024-10-16 14:00:03.070 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1460.824801303744], b:[2.57258492192251E12] 2024-10-16 14:00:03.075 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:03.170 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:03.200 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:03.226 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:00:03.336 INFO 4486 --- [subtask-executor32,2233273729644a0c,902d7cde91439e21] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934023, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 14:00:03.371 INFO 4486 --- [ocp-async-8,a0ae04081cb326fc,29b6e7988f4f12f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:03.574 INFO 4486 --- [ocp-schedules-17,5b03850c0497e0b4,ef4cdcf3e2fc80f1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:00:03.575 INFO 4486 --- [ocp-schedules-17,5b03850c0497e0b4,ef4cdcf3e2fc80f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:03.576 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:00:03.576 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:00:03.576 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:00:04.459 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:04.994 INFO 4486 --- [prometheus-minute-schedule-1,fbd75bf3ad793158,809d42a7a30c5443] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:05.000 INFO 4486 --- [prometheus-minute-schedule-1,fbd75bf3ad793158,809d42a7a30c5443] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:05.372 INFO 4486 --- [ocp-async-4,83f98845ab600fc8,a9f521288afae51d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:05.552 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d49a22aeba57d741,d3d30b28316b3cc3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=d49a22aeba57d741, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 14:00:05.667 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,3e25bc6f3d7d7e4c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934029, operation:EXECUTE, state: RUNNING 2024-10-16 14:00:05.682 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7934029, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266497, task_operation=execute, latest_execution_start_time=2024-10-16T14:00:05.664+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=7934029}, listMap={}}, executor=10.197.49.184 2024-10-16 14:00:05.687 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 14:00:05.687 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 14:00:05.689 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.689 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:00:05.691 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.692 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.693 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.694 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 14:00:05.712 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.712 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.713 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.714 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 14:00:05.728 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.729 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.730 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.731 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 14:00:05.746 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.747 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.747 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.748 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 14:00:05.762 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.763 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.764 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.765 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 14:00:05.766 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d49a22aeba57d741,d3d30b28316b3cc3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=d49a22aeba57d741, duration=227 ms] 2024-10-16 14:00:05.774 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.775 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.775 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.776 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 14:00:05.781 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.782 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.783 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.783 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 14:00:05.789 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.789 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.790 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.791 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 14:00:05.798 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.799 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.800 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.800 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 14:00:05.807 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.808 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.809 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.810 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 14:00:05.817 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.817 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.818 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.819 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 14:00:05.829 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.830 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.831 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.831 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 14:00:05.846 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.847 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.848 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.848 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 14:00:05.854 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.856 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.857 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.858 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 14:00:05.865 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.866 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.867 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.867 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 14:00:05.875 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.875 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.876 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.877 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 14:00:05.882 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.883 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.884 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.884 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 14:00:05.891 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.892 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.893 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.893 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 14:00:05.899 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.900 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.900 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.901 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 14:00:05.907 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.907 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.908 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.909 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 14:00:05.914 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.915 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.916 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.916 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 14:00:05.922 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.923 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.923 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.924 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 14:00:05.929 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.929 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.930 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.931 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 14:00:05.935 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.936 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.937 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.937 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 14:00:05.942 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.943 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.944 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.944 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 14:00:05.950 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.950 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.951 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.952 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 14:00:05.957 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.958 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.959 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.959 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 14:00:05.965 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.966 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.967 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.967 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 14:00:05.973 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.973 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.974 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.975 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 14:00:05.980 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.981 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.981 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.982 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 14:00:05.987 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.987 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.988 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.988 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 14:00:05.995 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.996 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:05.996 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:05.997 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 14:00:06.003 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.003 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.004 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.005 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 14:00:06.011 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.011 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.012 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.013 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 14:00:06.021 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.022 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.023 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.023 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 14:00:06.031 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.032 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.032 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.033 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 14:00:06.040 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.041 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.041 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.042 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 14:00:06.047 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.048 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.049 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.049 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 14:00:06.054 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.055 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.056 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.056 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 14:00:06.061 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.062 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.063 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.063 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 14:00:06.068 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.068 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.070 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.070 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 14:00:06.075 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.075 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.076 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.076 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 14:00:06.087 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.087 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.088 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.089 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 14:00:06.110 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.110 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.111 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.112 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 14:00:06.118 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.119 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.120 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.120 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 14:00:06.127 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.127 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.128 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.128 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 14:00:06.136 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.136 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.137 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.137 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 14:00:06.141 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.141 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.142 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.143 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 14:00:06.147 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.147 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.148 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.149 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 14:00:06.153 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 14:00:06.153 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 14:00:06.161 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.162 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:00:06.164 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.164 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.165 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.166 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 14:00:06.172 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.172 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:00:06.173 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:06.173 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 14:00:06.179 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 14:00:06.187 INFO 4486 --- [subtask-executor32,2233273729644a0c,a0904bfdb5807508] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934029, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 14:00:07.066 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:00:07.066 INFO 4486 --- [ocp-schedules-13,0ef4345d3e6fc6e3,349a606ee2025ee4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:00:07.068 INFO 4486 --- [ocp-schedules-13,0ef4345d3e6fc6e3,349a606ee2025ee4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:07.081 INFO 4486 --- [ocp-schedules-5,6de745115995413a,99f9b15447144997] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:00:07.083 INFO 4486 --- [ocp-schedules-5,6de745115995413a,99f9b15447144997] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:00:07.089 INFO 4486 --- [ocp-schedules-24,95833a402f57ac8f,7ab67ec02939b161] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:00:07.089 INFO 4486 --- [ocp-schedules-22,e3a8a0b429e42f5c,0eb2b95943c27269] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:00:07.090 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:00:07.090 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:00:07.090 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:00:07.090 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:00:07.091 INFO 4486 --- [host-check-agent-status1,e3a8a0b429e42f5c,c5e06cf77e0eec97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:00:07.091 INFO 4486 --- [host-check-agent-status2,e3a8a0b429e42f5c,a2bb0c52d6e3bcf9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:00:07.091 INFO 4486 --- [host-check-agent-status3,e3a8a0b429e42f5c,80cc52ec0b6ea873] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:00:07.091 INFO 4486 --- [ocp-schedules-7,6afb4b9003eb3b96,0064e8fae71b9960] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:00:07.091 INFO 4486 --- [host-check-agent-status4,e3a8a0b429e42f5c,8ce72c51d1c11522] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:00:07.091 INFO 4486 --- [host-check-clockdiff2,6afb4b9003eb3b96,6f88ff10bbe92cb4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:00:07.092 INFO 4486 --- [host-check-clockdiff1,6afb4b9003eb3b96,ab467de8e6f3f40e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:00:07.093 INFO 4486 --- [host-check-clockdiff3,6afb4b9003eb3b96,d1d94a64f73bee1b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:00:07.093 INFO 4486 --- [host-check-clockdiff4,6afb4b9003eb3b96,b2d84674a562dfb2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:00:07.098 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:00:07.098 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:00:07.098 INFO 4486 --- [host-check-agent-status2,e3a8a0b429e42f5c,a2bb0c52d6e3bcf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:00:07.099 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:00:07.100 INFO 4486 --- [host-check-agent-status3,e3a8a0b429e42f5c,80cc52ec0b6ea873] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:00:07.100 INFO 4486 --- [host-check-agent-status4,e3a8a0b429e42f5c,8ce72c51d1c11522] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:00:07.112 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:07.114 INFO 4486 --- [host-check-agent-status1,e3a8a0b429e42f5c,c5e06cf77e0eec97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:07.115 INFO 4486 --- [host-check-agent-status2,e3a8a0b429e42f5c,a2bb0c52d6e3bcf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:00:07.118 INFO 4486 --- [host-check-agent-status3,e3a8a0b429e42f5c,80cc52ec0b6ea873] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:00:07.115 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:00:07.122 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:00:07.115 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:00:07.122 INFO 4486 --- [host-check-clockdiff3,6afb4b9003eb3b96,d1d94a64f73bee1b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058407 4 4 , error= 2024-10-16 14:00:07.122 INFO 4486 --- [host-check-clockdiff2,6afb4b9003eb3b96,6f88ff10bbe92cb4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058407 4 4 , error= 2024-10-16 14:00:07.126 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:07.126 INFO 4486 --- [host-check-agent-status1,e3a8a0b429e42f5c,c5e06cf77e0eec97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:07.127 INFO 4486 --- [host-check-clockdiff1,6afb4b9003eb3b96,ab467de8e6f3f40e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058407 0 0 , error= 2024-10-16 14:00:07.131 INFO 4486 --- [host-check-agent-status4,e3a8a0b429e42f5c,8ce72c51d1c11522] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:00:07.133 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.133 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:07.134 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:00:07.155 INFO 4486 --- [host-check-agent-status2,e3a8a0b429e42f5c,a2bb0c52d6e3bcf9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:00:07.155 INFO 4486 --- [host-check-agent-status1,e3a8a0b429e42f5c,c5e06cf77e0eec97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:00:07.155 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.157 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:07.157 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:00:07.157 INFO 4486 --- [host-check-agent-status3,e3a8a0b429e42f5c,80cc52ec0b6ea873] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:00:07.158 INFO 4486 --- [host-check-agent-status4,e3a8a0b429e42f5c,8ce72c51d1c11522] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:00:07.158 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:00:07.160 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:00:07.161 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:00:07.164866856+08:00 2024-10-16 14:00:07.161 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:00:07.161 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:00:07.163675596+08:00 2024-10-16 14:00:07.161 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:00:07.161+08:00 2024-10-16 14:00:07.161 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:00:07.162 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:00:07.161+08:00 2024-10-16 14:00:07.162 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:00:07.162121595+08:00 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:00:07.166169521+08:00 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status1,e3a8a0b429e42f5c,c5e06cf77e0eec97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status1,e3a8a0b429e42f5c,c5e06cf77e0eec97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status1,e3a8a0b429e42f5c,c5e06cf77e0eec97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:00:07.162+08:00 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status2,e3a8a0b429e42f5c,a2bb0c52d6e3bcf9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status2,e3a8a0b429e42f5c,a2bb0c52d6e3bcf9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status2,e3a8a0b429e42f5c,a2bb0c52d6e3bcf9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status3,e3a8a0b429e42f5c,80cc52ec0b6ea873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status3,e3a8a0b429e42f5c,80cc52ec0b6ea873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status3,e3a8a0b429e42f5c,80cc52ec0b6ea873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:00:07.163+08:00 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status4,e3a8a0b429e42f5c,8ce72c51d1c11522] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status4,e3a8a0b429e42f5c,8ce72c51d1c11522] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:00:07.163 INFO 4486 --- [host-check-agent-status4,e3a8a0b429e42f5c,8ce72c51d1c11522] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:00:07.164 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:00:07.161+08:00 and status AVAILABLE 2024-10-16 14:00:07.165 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:00:07.161+08:00 and status AVAILABLE 2024-10-16 14:00:07.165 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:00:07.163+08:00 and status AVAILABLE 2024-10-16 14:00:07.165 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:00:07.162+08:00 and status AVAILABLE 2024-10-16 14:00:07.167 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:00:07.168 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:00:07.168 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:00:07.168 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:00:07.171 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.175 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:00:07.175 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:00:07.177 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:00:07.177 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:00:07.178 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:00:07.178 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:07.179 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:00:07.180 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:07.180 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:00:07.184 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.185 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:07.194 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.195 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:07.204 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:07.205 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.205 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:07.206 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:07.206 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:07.207 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:07.210 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.212 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:00:07.223 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.224 INFO 4486 --- [ocp-schedules-29,a38ac00c36f65858,655b79b2b57b7043] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:00:07.233 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:00:07.242 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:00:07.242 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:00:07.246 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:00:07.253 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:00:07.253 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:00:07.262 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:07.269 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:07.270 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:07.270 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:00:07.281 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:00:07.291 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:00:07.296 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:07.302 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:00:07.307 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:00:07.313 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:00:07.317 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:00:07.319 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:07.323 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:00:07.335 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:07.340 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:07.347 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:00:07.349 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:00:07.357 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:00:07.358 INFO 4486 --- [host-check-host-status3,95833a402f57ac8f,36146cb6f844bafd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:00:07.372 INFO 4486 --- [ocp-async-7,8d9726dc56c1a5ab,2b9d88e78a80fa31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:07.372 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:07.379 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:00:07.390 INFO 4486 --- [host-check-host-status4,95833a402f57ac8f,d103a3e5a9b2194a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:00:07.396 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:00:07.407 INFO 4486 --- [host-check-host-status2,95833a402f57ac8f,f6a431bb67aeca6c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:00:07.428 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:00:07.440 INFO 4486 --- [host-check-host-status1,95833a402f57ac8f,b35e928165c73fb4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:00:07.459 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:07.572 INFO 4486 --- [ocp-schedules-16,deb8f939ced5587c,95f49a1cbcec689a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:00:07.572 INFO 4486 --- [ocp-schedules-16,dc593955791a6bcd,afef584bd5a99cf7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:00:07.574 INFO 4486 --- [ocp-schedules-16,dc593955791a6bcd,afef584bd5a99cf7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:07.604 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.604 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:07.618 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.619 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:07.640 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:07.641 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.642 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:00:07.644 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.645 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:07.671 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:07.672 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.672 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:07.673 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:00:07.674 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.677 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:00:07.678 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:07.686 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:07.686 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:07.697 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:07.708 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:07.709 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.710 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:07.711 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:00:07.717 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:00:07.723 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:07.725 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:07.753 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:00:07.789 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:07.790 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.791 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:00:07.818 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:00:07.819 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:00:07.829 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:00:07.839 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:07.841 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.841 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:00:07.848 INFO 4486 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:5266497, status:SUCCESS 2024-10-16 14:00:07.849 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:07.850 INFO 4486 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:5266497, status:SUCCESS 2024-10-16 14:00:07.850 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.851 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:00:07.851 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:07.851 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:00:07.855 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:07.858 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:00:07.863 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:07.894 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:07.894 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:07.927 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:07.927 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:07.957 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:07.957 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:07.985 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:07.985 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:07.994 INFO 4486 --- [prometheus-minute-schedule-1,204521d1b95fa481,dfc135b73c593c1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:07.999 INFO 4486 --- [prometheus-minute-schedule-1,204521d1b95fa481,dfc135b73c593c1d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:08.015 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:08.015 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:08.060 INFO 4486 --- [host-check-clockdiff4,6afb4b9003eb3b96,b2d84674a562dfb2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058408 4 4 , error= 2024-10-16 14:00:08.064 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:08.100 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:08.101 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:08.102 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:08.106 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:00:08.112 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:00:08.117 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:00:08.119 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:00:08.142 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:00:08.308 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:00:08.329 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cad706ee72ad9697,b6b81af47d2c6175] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cad706ee72ad9697, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:08.330 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,cad706ee72ad9697,b6b81af47d2c6175] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cad706ee72ad9697, duration=2 ms] 2024-10-16 14:00:08.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:08.337 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:08.338 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:00:08.370 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:00:08.370 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:00:09.372 INFO 4486 --- [ocp-async-10,b321cfc0fdf39266,c99bd987bddec5ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:10.006 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:00:10.007 INFO 4486 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:00:10.026 INFO 4486 --- [Check obproxy2,c8c134b195d5567c,ca28a5fa8dd5beeb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:00:10.027 INFO 4486 --- [Check obproxy1,9e007ac74d74d8d8,f35279765a69c1af] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:00:10.027 INFO 4486 --- [Check obproxy2,c8c134b195d5567c,ca28a5fa8dd5beeb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:10.027 INFO 4486 --- [Check obproxy1,9e007ac74d74d8d8,f35279765a69c1af] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:10.031 INFO 4486 --- [Collect obproxy2,5bc6583b965b4f19,6db732480e7b5c69] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:00:10.031 INFO 4486 --- [Collect obproxy2,5bc6583b965b4f19,6db732480e7b5c69] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:10.034 INFO 4486 --- [Collect obproxy1,6edda99d9b91aecd,ff1329e40d8bce8e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:00:10.034 INFO 4486 --- [Collect obproxy1,6edda99d9b91aecd,ff1329e40d8bce8e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:10.054 INFO 4486 --- [Collect obproxy2,5bc6583b965b4f19,6db732480e7b5c69] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:00:10.059 INFO 4486 --- [Collect obproxy2,5bc6583b965b4f19,6db732480e7b5c69] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:00:10.059 INFO 4486 --- [Check obproxy2,c8c134b195d5567c,ca28a5fa8dd5beeb] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:00:10.060 INFO 4486 --- [Check obproxy2,c8c134b195d5567c,ca28a5fa8dd5beeb] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:00:10.065 INFO 4486 --- [Collect obproxy2,5bc6583b965b4f19,6db732480e7b5c69] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:00:10.066 INFO 4486 --- [Collect obproxy1,6edda99d9b91aecd,ff1329e40d8bce8e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:00:10.071 INFO 4486 --- [Collect obproxy1,6edda99d9b91aecd,ff1329e40d8bce8e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:00:10.075 INFO 4486 --- [Check obproxy1,9e007ac74d74d8d8,f35279765a69c1af] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:00:10.075 INFO 4486 --- [Check obproxy1,9e007ac74d74d8d8,f35279765a69c1af] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:00:10.076 INFO 4486 --- [Collect obproxy1,6edda99d9b91aecd,ff1329e40d8bce8e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:00:10.142 INFO 4486 --- [ocp-schedules-14,db396b0aea758888,ace13e2c461fb34f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:00:10.190 WARN 4486 --- [scanner-13316898-1,db396b0aea758888,6d4e0131d06fe8f5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:00:10.232 INFO 4486 --- [alarm-detect-19,db396b0aea758888,d40a5e436bacd346] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058385 ~ 1729058400 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058400, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058400, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058400=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:00:10.348 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8dab9a05143ddce4,4cf45b46f3329283] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8dab9a05143ddce4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:10.348 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8dab9a05143ddce4,4cf45b46f3329283] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8dab9a05143ddce4, duration=2 ms] 2024-10-16 14:00:10.460 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:11.011 INFO 4486 --- [prometheus-minute-schedule-1,3c5e627197798bb2,a43a1ca530d639d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:11.022 INFO 4486 --- [prometheus-minute-schedule-1,3c5e627197798bb2,a43a1ca530d639d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:11.031 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:00:11.226 INFO 4486 --- [ocp-schedules-15,18f75a0105b4688a,38c22fbfe7e9b699] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:00:11.372 INFO 4486 --- [ocp-async-5,4a84070d765603d8,02d04ee706abcf9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:11.882 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,fd63c5a86d23c33a,097fff6cea999dee] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fd63c5a86d23c33a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:11.882 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,fd63c5a86d23c33a,097fff6cea999dee] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fd63c5a86d23c33a, duration=1 ms] 2024-10-16 14:00:12.195 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5540e440177637e0,293826b20c0b964a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5540e440177637e0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:12.195 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5540e440177637e0,293826b20c0b964a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5540e440177637e0, duration=1 ms] 2024-10-16 14:00:13.026 INFO 4486 --- [ocp-async-8,d727d19de30225f6,ea24fcf4615f2a8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:00:13.373 INFO 4486 --- [ocp-async-7,14d2f562fa8b0e04,fef43c1123da1234] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:13.461 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:13.577 INFO 4486 --- [ocp-schedules-17,5b03850c0497e0b4,ef4cdcf3e2fc80f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:00:13.994 INFO 4486 --- [prometheus-minute-schedule-1,2e84e8bf06301c97,2eecf3d994a0b7a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:14.004 INFO 4486 --- [prometheus-minute-schedule-1,2e84e8bf06301c97,2eecf3d994a0b7a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:14.502 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058340, nodeCount=290, affectRows=290 2024-10-16 14:00:14.503 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058340, nodeCount=232, affectRows=232 2024-10-16 14:00:14.504 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058340, nodeCount=290, validCount=290 2024-10-16 14:00:14.505 INFO 4486 --- [metric-archive-second-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058340, nodeCount=231, validCount=231 2024-10-16 14:00:14.518 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058340, nodeCount=376, affectRows=376 2024-10-16 14:00:14.519 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058340, nodeCount=552, affectRows=552 2024-10-16 14:00:14.520 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058340, nodeCount=375, validCount=375 2024-10-16 14:00:14.521 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058340, nodeCount=552, validCount=552 2024-10-16 14:00:14.532 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058340, nodeCount=1000, affectRows=1000 2024-10-16 14:00:14.537 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058340, nodeCount=1000, validCount=1000 2024-10-16 14:00:14.538 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058340, nodeCount=1000, affectRows=1000 2024-10-16 14:00:14.542 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058340, nodeCount=999, validCount=999 2024-10-16 14:00:15.373 INFO 4486 --- [ocp-async-10,60c7899a621660b2,5280420bb9816557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:16.066 INFO 4486 --- [ocp-schedules-15,63a5aba5a2310cc1,88e0bee57d406cb5] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 14:00:16.066 INFO 4486 --- [ocp-schedules-15,63a5aba5a2310cc1,88e0bee57d406cb5] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 14:00:16.099 INFO 4486 --- [pool-283-thread-1,63a5aba5a2310cc1,3ad07c11e5910afb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:00:16.099 INFO 4486 --- [pool-283-thread-3,63a5aba5a2310cc1,c08453c4a60e44c0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:00:16.100 INFO 4486 --- [pool-283-thread-2,63a5aba5a2310cc1,3903103882c83893] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:00:16.125 INFO 4486 --- [pool-283-thread-1,63a5aba5a2310cc1,3ad07c11e5910afb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:16.126 INFO 4486 --- [pool-283-thread-2,63a5aba5a2310cc1,3903103882c83893] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:16.129 INFO 4486 --- [pool-283-thread-1,63a5aba5a2310cc1,3ad07c11e5910afb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058416000000] 2024-10-16 14:00:16.131 INFO 4486 --- [pool-283-thread-2,63a5aba5a2310cc1,3903103882c83893] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058416000000] 2024-10-16 14:00:16.134 INFO 4486 --- [pool-283-thread-3,63a5aba5a2310cc1,c08453c4a60e44c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:16.139 INFO 4486 --- [pool-283-thread-3,63a5aba5a2310cc1,c08453c4a60e44c0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058416000000] 2024-10-16 14:00:16.412 WARN 4486 --- [pool-283-thread-3,63a5aba5a2310cc1,c08453c4a60e44c0] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 14:00:16.412 WARN 4486 --- [pool-283-thread-3,63a5aba5a2310cc1,c08453c4a60e44c0] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 14:00:16.415 INFO 4486 --- [pool-283-thread-3,63a5aba5a2310cc1,c08453c4a60e44c0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:00:16.462 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:16.703 WARN 4486 --- [pool-283-thread-1,63a5aba5a2310cc1,3ad07c11e5910afb] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 14:00:16.703 WARN 4486 --- [pool-283-thread-1,63a5aba5a2310cc1,3ad07c11e5910afb] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1009 2024-10-16 14:00:16.705 INFO 4486 --- [pool-283-thread-1,63a5aba5a2310cc1,3ad07c11e5910afb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:00:16.994 INFO 4486 --- [prometheus-minute-schedule-1,8fa09e9dffd07061,c6245abc2088ff66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:16.999 INFO 4486 --- [prometheus-minute-schedule-1,8fa09e9dffd07061,c6245abc2088ff66] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:17.003 WARN 4486 --- [pool-283-thread-2,63a5aba5a2310cc1,3903103882c83893] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 14:00:17.006 INFO 4486 --- [pool-283-thread-2,63a5aba5a2310cc1,3903103882c83893] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:00:17.009 INFO 4486 --- [ocp-schedules-15,63a5aba5a2310cc1,88e0bee57d406cb5] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 14:00:17.069 INFO 4486 --- [ocp-schedules-13,0ef4345d3e6fc6e3,349a606ee2025ee4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:00:17.162 INFO 4486 --- [ocp-schedules-5,6de745115995413a,99f9b15447144997] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:00:17.271 INFO 4486 --- [ocp-schedules-7,5da62e10cc22f772,6d9982e12a0ffb8a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:00:17.373 INFO 4486 --- [ocp-async-8,be595b17cbaf25f6,4e5eccd033417fef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:18.335 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d99aca3dcf92114b,6470ed8fb9581f3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d99aca3dcf92114b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:18.335 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d99aca3dcf92114b,6470ed8fb9581f3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d99aca3dcf92114b, duration=2 ms] 2024-10-16 14:00:19.246 INFO 4486 --- [ocp-schedules-29,bed585596bfabc12,fd2a23580d0e6e7b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:00:19.374 INFO 4486 --- [ocp-async-1,e1bdf5e3ddc72514,6e96b444ba3b9165] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:19.463 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:19.994 INFO 4486 --- [prometheus-minute-schedule-1,0ae098f0dfde2446,810b585e748b5e95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:20.000 INFO 4486 --- [prometheus-minute-schedule-1,0ae098f0dfde2446,810b585e748b5e95] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:20.360 INFO 4486 --- [ocp-schedules-5,1f64e1c7bb7f882f,f61fb4ecb21efd9e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:00:20.360 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f2005842a3804a31,d5f539b85a26eaf7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f2005842a3804a31, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:20.360 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,f2005842a3804a31,d5f539b85a26eaf7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f2005842a3804a31, duration=2 ms] 2024-10-16 14:00:20.400 WARN 4486 --- [scanner-13317169-1,1f64e1c7bb7f882f,2e670f40a99b2423] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:00:20.813 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 14:00:20.996 INFO 4486 --- [prometheus-cache-monitor1,9625e7504e296aa2,ce85880eb63dba5f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13859,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":13126} 2024-10-16 14:00:20.996 INFO 4486 --- [prometheus-cache-monitor1,9625e7504e296aa2,ce85880eb63dba5f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:00:21.231 INFO 4486 --- [ocp-schedules-15,3e9c26ca7d33542a,91bde6c937e1d209] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:00:21.374 INFO 4486 --- [ocp-async-2,b68bb637610f25de,6b7b825983f411df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:21.891 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,283cc320f95fdf99,7e4577a80c8d296e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=283cc320f95fdf99, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:21.891 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,283cc320f95fdf99,7e4577a80c8d296e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=283cc320f95fdf99, duration=2 ms] 2024-10-16 14:00:21.988 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,13532392ce50af7d,432158ba86250761] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=13532392ce50af7d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:00:21.988 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,13532392ce50af7d,432158ba86250761] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=13532392ce50af7d, duration=1 ms] 2024-10-16 14:00:22.199 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7756265ceae44940,05b12fc8064ce278] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7756265ceae44940, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:22.200 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7756265ceae44940,05b12fc8064ce278] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7756265ceae44940, duration=1 ms] 2024-10-16 14:00:22.464 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:22.994 INFO 4486 --- [prometheus-minute-schedule-1,b8168b922430633f,8d6e832ad0930503] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:23.000 INFO 4486 --- [prometheus-minute-schedule-1,b8168b922430633f,8d6e832ad0930503] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:23.026 INFO 4486 --- [ocp-async-7,3451d8bcb43563d6,5569007fe7c97326] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:00:23.374 INFO 4486 --- [ocp-async-5,4b2a5e175a7fd6b0,6b65bf5f937ec62a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:23.893 INFO 4486 --- [ocp-schedules-18,32c852906aeb5790,d7c69837e619bccb] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:00:25.181 INFO 4486 --- [ocp-schedules-22,45ab5cc610f0081a,9498c260ed8fbc98] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:00:25.374 INFO 4486 --- [ocp-async-1,bf996fb380bc3dfe,5afbc3d5d66223b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:25.466 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:25.994 INFO 4486 --- [prometheus-minute-schedule-1,73f66bef93daae89,52eef1d7e67297e8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:26.000 INFO 4486 --- [prometheus-minute-schedule-1,73f66bef93daae89,52eef1d7e67297e8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:26.687 INFO 4486 --- [ocp-schedules-24,76ee6ae5aef6bab5,49c5cc937fb07888] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:00:26.687 INFO 4486 --- [ocp-schedules-24,76ee6ae5aef6bab5,49c5cc937fb07888] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:00:26.726 INFO 4486 --- [Collect all tenants major compactions1,76ee6ae5aef6bab5,d11fbfa3d221f8e1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:00:26.726 INFO 4486 --- [Collect all tenants major compactions4,76ee6ae5aef6bab5,84971061f795f423] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:00:26.726 INFO 4486 --- [Collect all tenants major compactions5,76ee6ae5aef6bab5,e81b3a70c660427c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:00:26.726 INFO 4486 --- [Collect all tenants major compactions3,76ee6ae5aef6bab5,5f4367095ea7b403] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:00:26.727 INFO 4486 --- [Collect all tenants major compactions2,76ee6ae5aef6bab5,0cf46acc23678d9e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:00:26.740 INFO 4486 --- [Collect all tenants major compactions2,76ee6ae5aef6bab5,0cf46acc23678d9e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:26.741 INFO 4486 --- [Collect all tenants major compactions1,76ee6ae5aef6bab5,d11fbfa3d221f8e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:26.742 INFO 4486 --- [Collect all tenants major compactions2,76ee6ae5aef6bab5,0cf46acc23678d9e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:00:26.742 INFO 4486 --- [Collect all tenants major compactions1,76ee6ae5aef6bab5,d11fbfa3d221f8e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:00:26.746 INFO 4486 --- [Collect all tenants major compactions2,76ee6ae5aef6bab5,0cf46acc23678d9e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:00:26.746 INFO 4486 --- [Collect all tenants major compactions1,76ee6ae5aef6bab5,d11fbfa3d221f8e1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:00:26.746 INFO 4486 --- [Collect all tenants major compactions2,76ee6ae5aef6bab5,0cf46acc23678d9e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:00:26.746 INFO 4486 --- [Collect all tenants major compactions1,76ee6ae5aef6bab5,d11fbfa3d221f8e1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:00:26.746 INFO 4486 --- [Collect all tenants major compactions3,76ee6ae5aef6bab5,5f4367095ea7b403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:26.749 INFO 4486 --- [Collect all tenants major compactions3,76ee6ae5aef6bab5,5f4367095ea7b403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:00:26.754 INFO 4486 --- [Collect all tenants major compactions3,76ee6ae5aef6bab5,5f4367095ea7b403] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:00:26.754 INFO 4486 --- [Collect all tenants major compactions3,76ee6ae5aef6bab5,5f4367095ea7b403] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:00:26.759 INFO 4486 --- [Collect all tenants major compactions4,76ee6ae5aef6bab5,84971061f795f423] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:26.759 INFO 4486 --- [Collect all tenants major compactions5,76ee6ae5aef6bab5,e81b3a70c660427c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:26.760 INFO 4486 --- [Collect all tenants major compactions4,76ee6ae5aef6bab5,84971061f795f423] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:00:26.761 INFO 4486 --- [Collect all tenants major compactions5,76ee6ae5aef6bab5,e81b3a70c660427c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:00:26.764 INFO 4486 --- [Collect all tenants major compactions4,76ee6ae5aef6bab5,84971061f795f423] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:00:26.765 INFO 4486 --- [Collect all tenants major compactions4,76ee6ae5aef6bab5,84971061f795f423] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:00:26.765 INFO 4486 --- [Collect all tenants major compactions5,76ee6ae5aef6bab5,e81b3a70c660427c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:00:26.765 INFO 4486 --- [Collect all tenants major compactions5,76ee6ae5aef6bab5,e81b3a70c660427c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:00:26.766 INFO 4486 --- [ocp-schedules-24,76ee6ae5aef6bab5,49c5cc937fb07888] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:00:27.374 INFO 4486 --- [ocp-async-6,1606280ee41cc10c,989d37ba9c143b12] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:27.832 INFO 4486 --- [ocp-schedules-6,b1a453cee76425e0,f572dc6c7161ee83] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:00:27.832+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:00:27.834 INFO 4486 --- [ocp-schedules-6,b1a453cee76425e0,f572dc6c7161ee83] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:00:27.835 INFO 4486 --- [ocp-schedules-6,b1a453cee76425e0,f572dc6c7161ee83] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:00:27.835 INFO 4486 --- [ocp-schedules-6,b1a453cee76425e0,f572dc6c7161ee83] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:00:28.341 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a09a9cd12c93557c,a8e067ce50ee9ea6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a09a9cd12c93557c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:28.342 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a09a9cd12c93557c,a8e067ce50ee9ea6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a09a9cd12c93557c, duration=3 ms] 2024-10-16 14:00:28.467 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:28.766 INFO 4486 --- [ocp-schedules-5,5213cc12084fb15c,901d0fff9693e11b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 14:00:28.768 INFO 4486 --- [ocp-schedules-5,5213cc12084fb15c,901d0fff9693e11b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:28.804 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:00:28.804 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.808 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:00:28.818 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.819 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:00:28.823 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:00:28.824 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.829 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:00:28.830 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.830 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:00:28.843 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.843 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.844 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:00:28.844 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:00:28.863 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.863 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:00:28.863 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:00:28.864 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.867 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.868 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:00:28.870 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:00:28.882 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.883 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:00:28.884 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.884 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:28.896 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.898 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:28.900 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.901 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:00:28.902 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.903 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:28.915 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.916 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.916 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:28.917 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:00:28.918 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.918 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:00:28.921 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.922 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:00:28.931 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.932 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:00:28.934 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.935 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:00:28.937 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.938 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:00:28.947 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.948 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:00:28.951 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.952 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:00:28.954 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.955 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:28.966 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.967 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:28.968 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.969 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:28.969 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.969 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.970 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:28.970 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:28.971 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.971 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:28.972 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.973 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:28.973 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.974 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:28.975 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.975 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.976 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:28.977 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:28.978 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.979 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:28.979 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.980 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:28.980 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.981 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.981 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:28.982 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:28.984 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.985 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.985 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:28.986 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.986 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.986 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:28.987 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:28.987 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:28.987 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 14:00:28.989 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.990 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:28.993 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:28.994 INFO 4486 --- [prometheus-minute-schedule-1,8148d272ddf29fa0,0b64b76f55adaf5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:28.995 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:00:29.000 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:29.000 INFO 4486 --- [prometheus-minute-schedule-1,8148d272ddf29fa0,0b64b76f55adaf5f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:29.004 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.007 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:29.012 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.013 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:29.015 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.017 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:29.019 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.020 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:29.021 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:29.022 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.022 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:00:29.023 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 14:00:29.026 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:00:29.033 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.035 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:29.039 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.040 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 14:00:29.043 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.044 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:29.047 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.047 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 14:00:29.048 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:00:29.048 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:29.052 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.053 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:29.056 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.057 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:29.061 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.062 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:00:29.066 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.066 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:29.067 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:00:29.068 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.069 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:00:29.070 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.072 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:00:29.074 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 14:00:29.078 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:00:29.081 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.081 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 14:00:29.081 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.081 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 14:00:29.081 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.081 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 14:00:29.086 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 14:00:29.086 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 14:00:29.145 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:29.147 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:29.148 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:00:29.157 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.157 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 14:00:29.157 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.157 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 14:00:29.157 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:00:29.157 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 14:00:29.161 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 14:00:29.161 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:00:29.375 INFO 4486 --- [ocp-async-3,d22f67ca080c0252,4f003a49ed249f1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:30.103 INFO 4486 --- [prometheus-second-schedule-1,beb024952399d489,9183811c4a1551f4] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:00:30.367 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b20426feaba47245,c531d78ea0a4227f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b20426feaba47245, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:30.367 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b20426feaba47245,c531d78ea0a4227f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b20426feaba47245, duration=1 ms] 2024-10-16 14:00:30.609 INFO 4486 --- [ocp-schedules-19,5ea9b4e4e9a08165,39973c8af56964af] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:00:30.640 WARN 4486 --- [scanner-13317468-1,5ea9b4e4e9a08165,e57ff6d943cab48a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:00:30.699 INFO 4486 --- [alarm-detect-18,5ea9b4e4e9a08165,451230dce28ac507] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058400 ~ 1729058420 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058415, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058415, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058415=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:00:30.700 INFO 4486 --- [alarm-detect-18,5ea9b4e4e9a08165,451230dce28ac507] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 14:00:31.237 INFO 4486 --- [ocp-schedules-14,5a39713df5db86d3,811ddc414648269a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:00:31.375 INFO 4486 --- [ocp-async-4,6ca65bccd7b56ad9,3ee92138095f4a77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:31.468 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:31.896 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,af77e9aa32bb2ea0,b373ebf70e5206e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=af77e9aa32bb2ea0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:31.896 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,af77e9aa32bb2ea0,b373ebf70e5206e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=af77e9aa32bb2ea0, duration=1 ms] 2024-10-16 14:00:31.994 INFO 4486 --- [prometheus-minute-schedule-1,58cd387d2f69a8ea,35b4dc0b31671492] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:31.999 INFO 4486 --- [prometheus-minute-schedule-1,58cd387d2f69a8ea,35b4dc0b31671492] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:32.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ecab6c7d81810fc0,2cab2c55c843e324] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ecab6c7d81810fc0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:32.206 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,ecab6c7d81810fc0,2cab2c55c843e324] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ecab6c7d81810fc0, duration=2 ms] 2024-10-16 14:00:33.026 INFO 4486 --- [ocp-async-5,e3bf16c047876225,7bf0e9ceb3e7e635] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:00:33.375 INFO 4486 --- [ocp-async-7,5a7c87317cb6528a,8aeee644ebe8b0e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:34.469 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:34.994 INFO 4486 --- [prometheus-minute-schedule-1,ce3e9995d727cb25,6097de8ac3694452] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:35.001 INFO 4486 --- [prometheus-minute-schedule-1,ce3e9995d727cb25,6097de8ac3694452] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:35.375 INFO 4486 --- [ocp-async-10,97c94ac322abcfc2,81eb22bc6d556caa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:37.063 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:00:37.064 INFO 4486 --- [ocp-schedules-23,279a7c8088f6d1b4,26b253c13abc0cae] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:00:37.066 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:00:37.067 INFO 4486 --- [ocp-schedules-6,c84a6dde974d38b3,cdb27a2bd11b28f0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:00:37.068 INFO 4486 --- [ocp-schedules-6,c84a6dde974d38b3,cdb27a2bd11b28f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:37.069 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:00:37.070 INFO 4486 --- [ocp-schedules-23,279a7c8088f6d1b4,26b253c13abc0cae] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:00:37.081 INFO 4486 --- [ocp-schedules-7,bc5f40c5dfbc5ab5,9076c139cc21b8df] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:00:37.085 INFO 4486 --- [ocp-schedules-7,bc5f40c5dfbc5ab5,9076c139cc21b8df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:37.086 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:00:37.089 INFO 4486 --- [ocp-schedules-27,fdb34a8e24477233,e0be1d7111e1fd61] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:00:37.090 INFO 4486 --- [ocp-schedules-22,ce27e733158407a8,ec38bd65b8816d85] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:00:37.090 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:00:37.090 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:00:37.091 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:00:37.092 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:00:37.092 INFO 4486 --- [host-check-clockdiff1,ce27e733158407a8,9a20d3dd84a3c724] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:00:37.092 INFO 4486 --- [host-check-clockdiff2,ce27e733158407a8,cd65b7c4b11ff157] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:00:37.093 INFO 4486 --- [host-check-clockdiff3,ce27e733158407a8,daba3363206be07b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:00:37.093 INFO 4486 --- [host-check-clockdiff4,ce27e733158407a8,25fdcbcf25a20eb9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:00:37.095 INFO 4486 --- [ocp-schedules-4,4a914020c2f3f969,152e165aed0156dc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:00:37.096 INFO 4486 --- [host-check-agent-status1,4a914020c2f3f969,15469dd17f04ee97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:00:37.096 INFO 4486 --- [host-check-agent-status2,4a914020c2f3f969,4783ded73da12ecc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:00:37.098 INFO 4486 --- [host-check-agent-status3,4a914020c2f3f969,e224889cbe817de5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:00:37.098 INFO 4486 --- [host-check-agent-status4,4a914020c2f3f969,1dc24339b0fca3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:00:37.104 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:00:37.104 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:37.105 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:00:37.138 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:00:37.138 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:00:37.138 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:00:37.138 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:37.139 INFO 4486 --- [host-check-clockdiff1,ce27e733158407a8,9a20d3dd84a3c724] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058437 0 0 , error= 2024-10-16 14:00:37.147 INFO 4486 --- [host-check-agent-status1,4a914020c2f3f969,15469dd17f04ee97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:37.147 INFO 4486 --- [host-check-clockdiff4,ce27e733158407a8,25fdcbcf25a20eb9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058437 4 4 , error= 2024-10-16 14:00:37.147 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:00:37.148 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:37.148 INFO 4486 --- [host-check-agent-status2,4a914020c2f3f969,4783ded73da12ecc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:00:37.148 INFO 4486 --- [host-check-agent-status4,4a914020c2f3f969,1dc24339b0fca3d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:00:37.149 INFO 4486 --- [host-check-agent-status1,4a914020c2f3f969,15469dd17f04ee97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:37.149 INFO 4486 --- [host-check-agent-status3,4a914020c2f3f969,e224889cbe817de5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:00:37.150 INFO 4486 --- [host-check-agent-status4,4a914020c2f3f969,1dc24339b0fca3d4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:00:37.150 INFO 4486 --- [host-check-agent-status2,4a914020c2f3f969,4783ded73da12ecc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:00:37.151 INFO 4486 --- [host-check-agent-status3,4a914020c2f3f969,e224889cbe817de5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:00:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:37.154 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:00:37.168 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:37.171 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.173 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:00:37.175 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:00:37.175 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:00:37.177 INFO 4486 --- [host-check-agent-status1,4a914020c2f3f969,15469dd17f04ee97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:00:37.177 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:00:37.180773393+08:00 2024-10-16 14:00:37.177 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:00:37.178 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:00:37.177+08:00 2024-10-16 14:00:37.178 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:37.178 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:00:37.178 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:00:37.177918345+08:00 2024-10-16 14:00:37.178 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.179 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:00:37.178+08:00 2024-10-16 14:00:37.179 INFO 4486 --- [host-check-agent-status3,4a914020c2f3f969,e224889cbe817de5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:00:37.179 INFO 4486 --- [host-check-agent-status4,4a914020c2f3f969,1dc24339b0fca3d4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:00:37.179 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.183 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:00:37.183 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:00:37.184217221+08:00 2024-10-16 14:00:37.183 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:37.183 INFO 4486 --- [host-check-agent-status1,4a914020c2f3f969,15469dd17f04ee97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:00:37.183 INFO 4486 --- [host-check-agent-status1,4a914020c2f3f969,15469dd17f04ee97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:00:37.183 INFO 4486 --- [host-check-agent-status1,4a914020c2f3f969,15469dd17f04ee97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:00:37.184 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.184 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:00:37.177+08:00 and status AVAILABLE 2024-10-16 14:00:37.184 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.184 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:00:37.183+08:00 2024-10-16 14:00:37.184 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:00:37.188396893+08:00 2024-10-16 14:00:37.185 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:37.185 INFO 4486 --- [host-check-agent-status2,4a914020c2f3f969,4783ded73da12ecc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:00:37.185 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.185 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:00:37.184+08:00 2024-10-16 14:00:37.185 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:00:37.178+08:00 and status AVAILABLE 2024-10-16 14:00:37.186 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:00:37.186 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:00:37.186 INFO 4486 --- [host-check-agent-status3,4a914020c2f3f969,e224889cbe817de5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:00:37.186 INFO 4486 --- [host-check-agent-status3,4a914020c2f3f969,e224889cbe817de5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:00:37.186 INFO 4486 --- [host-check-agent-status3,4a914020c2f3f969,e224889cbe817de5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:00:37.188 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:00:37.188 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:00:37.183+08:00 and status AVAILABLE 2024-10-16 14:00:37.188 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:00:37.188 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:00:37.184+08:00 and status AVAILABLE 2024-10-16 14:00:37.188 INFO 4486 --- [host-check-agent-status2,4a914020c2f3f969,4783ded73da12ecc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:00:37.188 INFO 4486 --- [host-check-agent-status2,4a914020c2f3f969,4783ded73da12ecc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:00:37.188 INFO 4486 --- [host-check-agent-status2,4a914020c2f3f969,4783ded73da12ecc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:00:37.189 INFO 4486 --- [host-check-agent-status4,4a914020c2f3f969,1dc24339b0fca3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:00:37.189 INFO 4486 --- [host-check-agent-status4,4a914020c2f3f969,1dc24339b0fca3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:00:37.189 INFO 4486 --- [host-check-agent-status4,4a914020c2f3f969,1dc24339b0fca3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:00:37.190 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:00:37.190 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:00:37.195 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.195 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.195 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.195 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.195 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.196 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:00:37.196 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.196 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.197 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:37.197 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:00:37.197 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:37.199 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:00:37.199 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:37.200 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:00:37.200 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:00:37.202 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:00:37.210 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.211 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:00:37.213 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.214 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:00:37.216 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:00:37.221 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:37.223 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:37.225 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.225 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:37.226 INFO 4486 --- [ocp-schedules-29,b6cf0aa5d05f2b38,05603b4a1832d6c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:00:37.229 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:00:37.254 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:00:37.254 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:00:37.258 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:00:37.264 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:00:37.264 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:00:37.268 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:00:37.279 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:37.281 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:37.283 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:37.296 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:00:37.306 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:00:37.309 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:00:37.311 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:00:37.318 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:00:37.321 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:00:37.321 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:00:37.322 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:00:37.332 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:00:37.335 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:37.336 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:37.348 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:37.365 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:00:37.370 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:00:37.375 INFO 4486 --- [ocp-async-4,1c75ec2f5008a4bc,eae9a511cdc75401] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:37.376 INFO 4486 --- [host-check-host-status2,fdb34a8e24477233,df334659abfd7759] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:00:37.376 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:00:37.381 INFO 4486 --- [host-check-host-status3,fdb34a8e24477233,84b09ba23db4712f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:00:37.382 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:00:37.386 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:00:37.392 INFO 4486 --- [host-check-host-status4,fdb34a8e24477233,ef51760fb13bfd80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:00:37.402 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:00:37.467 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:00:37.470 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:37.471 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:37.474 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.475 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:00:37.477 INFO 4486 --- [host-check-host-status1,fdb34a8e24477233,9c3e358cfd31aab7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:00:37.480 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.481 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.482 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:00:37.494 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.494 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.494 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.494 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.494 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.495 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.521 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:00:37.567 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:00:37.662 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:37.677 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:37.679 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.680 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:00:37.681 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:00:37.681 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.682 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:00:37.683 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.684 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.685 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:00:37.693 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.693 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.693 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.693 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.693 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.693 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.701 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:00:37.891 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:00:37.936 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:37.938 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.939 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:00:37.940 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.941 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.942 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:00:37.953 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.953 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.953 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.953 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.953 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:37.954 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:37.964 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:00:37.974 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:00:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:00:37.994 INFO 4486 --- [prometheus-minute-schedule-1,833ba6ee672c59d5,91402463dd0ac36b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:38.000 INFO 4486 --- [prometheus-minute-schedule-1,833ba6ee672c59d5,91402463dd0ac36b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:38.050 INFO 4486 --- [host-check-clockdiff3,ce27e733158407a8,daba3363206be07b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058438 4 4 , error= 2024-10-16 14:00:38.062 INFO 4486 --- [host-check-clockdiff2,ce27e733158407a8,cd65b7c4b11ff157] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058438 4 4 , error= 2024-10-16 14:00:38.193 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:00:38.236 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.239 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.240 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:00:38.243 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:38.244 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.245 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:00:38.255 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:38.255 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:38.255 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:38.255 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:38.255 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:00:38.255 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.317 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:00:38.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a955c44581785ed2,193a890979731acb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a955c44581785ed2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:38.347 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,a955c44581785ed2,193a890979731acb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a955c44581785ed2, duration=1 ms] 2024-10-16 14:00:38.370 INFO 4486 --- [ocp-schedules-16,dc593955791a6bcd,afef584bd5a99cf7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:00:38.371 INFO 4486 --- [ocp-schedules-16,679c6ace7cee88b7,9b5bcd4229d825f2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:00:38.373 INFO 4486 --- [ocp-schedules-16,679c6ace7cee88b7,9b5bcd4229d825f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:00:38.398 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.399 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:38.413 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.414 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:38.430 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.431 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.431 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:00:38.435 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.436 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:38.461 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.462 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.462 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:38.463 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:00:38.464 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.467 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:00:38.468 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:38.476 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.477 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:38.485 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.493 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.494 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.494 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:38.495 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:00:38.499 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:00:38.505 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:38.506 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:38.517 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:00:38.321999, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:00:38.518 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:00:38.321999, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:00:38.526 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:00:38.538 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.541 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.542 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:00:38.561 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:00:38.321999} } 2024-10-16 14:00:38.565 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:00:38.565 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:00:38.588 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.590 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.591 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:00:38.594 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.596 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:00:38.597 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:00:38.601 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:38.606 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.606 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.607 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:00:38.616 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:00:38.616 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:00:38.619 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:00:38.626 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:00:38.627 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.627 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:38.628 INFO 4486 --- [ocp-schedules-30,3f1d50afb7dcf143,99be58fcd18bddbe] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:00:38.654 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.654 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:38.678 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.678 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:38.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:00:38.726 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.726 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:00:38.751 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:00:38.777 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.779 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.780 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:00:38.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:00:38.789 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:00:38.796 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:00:38.798 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:00:38.823 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:00:38.946 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:00:38.970 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:00:38.971 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:00:38.973 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:00:39.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:00:39.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:00:39.251 INFO 4486 --- [ocp-schedules-20,d684823454107125,97534f6ee430ea4b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:00:39.376 INFO 4486 --- [ocp-async-6,14c2e9d1f769ddc0,f6ca235075ec0e02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:40.006 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:00:40.006 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:00:40.027 INFO 4486 --- [Check obproxy1,542f6034fc07bce1,ec4198c39e5cb0f2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:00:40.027 INFO 4486 --- [Check obproxy1,542f6034fc07bce1,ec4198c39e5cb0f2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:40.027 INFO 4486 --- [Check obproxy2,d29d07697b3c1a63,d3b9a590a7de4598] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:00:40.027 INFO 4486 --- [Check obproxy2,d29d07697b3c1a63,d3b9a590a7de4598] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:40.033 INFO 4486 --- [Collect obproxy2,7047125010530d19,8135e6ec398cd78e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:00:40.033 INFO 4486 --- [Collect obproxy1,4d595810ece82d11,275f7faf72dff0e6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:00:40.033 INFO 4486 --- [Collect obproxy2,7047125010530d19,8135e6ec398cd78e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:40.033 INFO 4486 --- [Collect obproxy1,4d595810ece82d11,275f7faf72dff0e6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:00:40.051 INFO 4486 --- [Check obproxy1,542f6034fc07bce1,ec4198c39e5cb0f2] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:00:40.051 INFO 4486 --- [Check obproxy1,542f6034fc07bce1,ec4198c39e5cb0f2] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:00:40.053 INFO 4486 --- [Collect obproxy1,4d595810ece82d11,275f7faf72dff0e6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:00:40.053 INFO 4486 --- [Collect obproxy2,7047125010530d19,8135e6ec398cd78e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:00:40.056 INFO 4486 --- [Collect obproxy1,4d595810ece82d11,275f7faf72dff0e6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:00:40.057 INFO 4486 --- [Collect obproxy2,7047125010530d19,8135e6ec398cd78e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:00:40.059 INFO 4486 --- [Collect obproxy1,4d595810ece82d11,275f7faf72dff0e6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:00:40.060 INFO 4486 --- [Check obproxy2,d29d07697b3c1a63,d3b9a590a7de4598] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:00:40.060 INFO 4486 --- [Check obproxy2,d29d07697b3c1a63,d3b9a590a7de4598] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:00:40.062 INFO 4486 --- [Collect obproxy2,7047125010530d19,8135e6ec398cd78e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:00:40.372 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,92c80d6ba83a2acc,aa917814052dc186] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=92c80d6ba83a2acc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:40.373 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,92c80d6ba83a2acc,aa917814052dc186] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=92c80d6ba83a2acc, duration=2 ms] 2024-10-16 14:00:40.470 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:40.753 INFO 4486 --- [ocp-schedules-4,aaffeb708a129732,78cac20984c0d6f2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:00:40.786 WARN 4486 --- [scanner-13317699-1,aaffeb708a129732,488885382d1d7f54] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:00:40.835 INFO 4486 --- [alarm-detect-18,aaffeb708a129732,30cbe1e709765ba2] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058415 ~ 1729058430 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058430, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058430, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058430=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:00:40.994 INFO 4486 --- [prometheus-minute-schedule-1,a784c526a609d550,c9a02f2481b8aade] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:40.995 INFO 4486 --- [prometheus-hour-schedule-1,ffea29e392340abd,09b8af44aad1ae20] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:00:41.000 INFO 4486 --- [prometheus-minute-schedule-1,a784c526a609d550,c9a02f2481b8aade] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:41.242 INFO 4486 --- [ocp-schedules-27,7f78c467c96daa56,6704062f254ea7f7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:00:41.376 INFO 4486 --- [ocp-async-3,b6505a86ddd83990,6645bb7786a5f8ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:41.901 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8044b3545f8df1d1,785a6d8e0f01ddf7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8044b3545f8df1d1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:41.901 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,8044b3545f8df1d1,785a6d8e0f01ddf7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8044b3545f8df1d1, duration=2 ms] 2024-10-16 14:00:42.182 INFO 4486 --- [ocp-schedules-3,861574f712a4c580,52a45e47ad582515] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:00:42.209 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,dee4eb25c0d7bcd8,376fccd2118d3ba0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dee4eb25c0d7bcd8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:42.209 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,dee4eb25c0d7bcd8,376fccd2118d3ba0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dee4eb25c0d7bcd8, duration=1 ms] 2024-10-16 14:00:43.027 INFO 4486 --- [ocp-async-10,61d45c2933b06813,d382caa99516902b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:00:43.376 INFO 4486 --- [ocp-async-1,076364120ce39bff,39c51528d3177064] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:43.471 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:43.994 INFO 4486 --- [prometheus-minute-schedule-1,4f194bbc6b2052bb,a06e84add18ca0c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:44.000 INFO 4486 --- [prometheus-minute-schedule-1,4f194bbc6b2052bb,a06e84add18ca0c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:45.061 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,bee2658659d339c1,ecef473c09751478] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=bee2658659d339c1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:00:45.061 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,bee2658659d339c1,ecef473c09751478] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=bee2658659d339c1, duration=1 ms] 2024-10-16 14:00:45.377 INFO 4486 --- [ocp-async-8,c0493bc7dd9002fc,88d02a8bb243dbee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:45.590 INFO 4486 --- [ocp-schedules-14,f72988051ddcff9b,4706028eb9215ac2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:00:45.594 INFO 4486 --- [ocp-schedules-14,f72988051ddcff9b,4706028eb9215ac2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:00:45.594 INFO 4486 --- [ocp-schedules-14,f72988051ddcff9b,4706028eb9215ac2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:00:45.625 INFO 4486 --- [ocp-schedules-28,71489ce781a533c6,0dae2e34c51e9a95] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:00:45.625 INFO 4486 --- [ocp-schedules-28,71489ce781a533c6,0dae2e34c51e9a95] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:00:46.473 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:46.994 INFO 4486 --- [prometheus-minute-schedule-1,2add9d47f7331466,eef1163310b7bf81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:46.999 INFO 4486 --- [prometheus-minute-schedule-1,2add9d47f7331466,eef1163310b7bf81] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:47.070 INFO 4486 --- [ocp-schedules-6,c84a6dde974d38b3,cdb27a2bd11b28f0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:00:47.188 INFO 4486 --- [ocp-schedules-7,bc5f40c5dfbc5ab5,9076c139cc21b8df] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:00:47.274 INFO 4486 --- [ocp-schedules-1,a236d85789969121,082aadacdbe59e58] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:00:47.377 INFO 4486 --- [ocp-async-10,c5ceb0bd37c1b612,774dfc4778d3fff0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:48.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ba61459769efee4e,f1d94b731a8b7669] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ba61459769efee4e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:48.352 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,ba61459769efee4e,f1d94b731a8b7669] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ba61459769efee4e, duration=2 ms] 2024-10-16 14:00:49.377 INFO 4486 --- [ocp-async-2,846c2bb6bfcfa503,d1a3ce20fd414a01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:49.473 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:49.994 INFO 4486 --- [prometheus-minute-schedule-1,6f45a5f023be9b34,170f4e4216819290] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:49.999 INFO 4486 --- [prometheus-minute-schedule-1,6f45a5f023be9b34,170f4e4216819290] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:50.378 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ded67c0f81ad3be5,af8663eb3d3912c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ded67c0f81ad3be5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:50.378 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ded67c0f81ad3be5,af8663eb3d3912c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ded67c0f81ad3be5, duration=1 ms] 2024-10-16 14:00:50.562 INFO 4486 --- [ocp-schedules-26,cd50329dc89dcbc3,0da7069aa6bf63de] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:00:50.562 INFO 4486 --- [ocp-schedules-26,cd50329dc89dcbc3,0da7069aa6bf63de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:00:50.564 INFO 4486 --- [ocp-schedules-26,cd50329dc89dcbc3,0da7069aa6bf63de] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:00:50.564 INFO 4486 --- [ocp-schedules-26,cd50329dc89dcbc3,0da7069aa6bf63de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:00:50.564 INFO 4486 --- [ocp-schedules-26,cd50329dc89dcbc3,0da7069aa6bf63de] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:00:50.564 INFO 4486 --- [ocp-schedules-26,cd50329dc89dcbc3,0da7069aa6bf63de] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 14:00:50.791 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0e7d958ab55e58ab,82dd2d163b9f9966] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=0e7d958ab55e58ab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:00:50.791 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0e7d958ab55e58ab,82dd2d163b9f9966] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=0e7d958ab55e58ab, duration=1 ms] 2024-10-16 14:00:50.861 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 14:00:50.861 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 14:00:50.875 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 14:00:50.875 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 14:00:50.875 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 14:00:50.886 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 14:00:50.887 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 14:00:50.887 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 14:00:50.892 INFO 4486 --- [ocp-schedules-2,609d8f44599238c5,799bd8896133915f] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 14:00:50.930 INFO 4486 --- [ocp-schedules-21,d6a104ce841729f4,21602bba2d91875b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:00:50.965 WARN 4486 --- [scanner-13318026-1,d6a104ce841729f4,864303c275a54bc7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:00:51.247 INFO 4486 --- [ocp-schedules-14,dc599f28953fa49e,e1357c59c8b39609] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:00:51.377 INFO 4486 --- [ocp-async-8,3c5d2b8833bd77bc,406360c753131331] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:51.436 INFO 4486 --- [ocp-schedules-10,b9404d15959ffc80,b2cf8cc7b17a3370] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:00:51.436 INFO 4486 --- [ocp-schedules-10,b9404d15959ffc80,b2cf8cc7b17a3370] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:00:51.436 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:00:51.436 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:00:51.436 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:00:51.436 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:00:51.436 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:00:51.442 INFO 4486 --- [ocp-schedules-10,b9404d15959ffc80,b2cf8cc7b17a3370] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:00:51.442 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:00:51.443 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:00:51.444 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:00:51.444 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:00:51.444 INFO 4486 --- [ocp-schedules-10,b9404d15959ffc80,b2cf8cc7b17a3370] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:00:51.444 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:00:51.444 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:00:51.445 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:00:51.446 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:00:51.446 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:00:51.447 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:00:51.468 INFO 4486 --- [ocp-schedules-10,b9404d15959ffc80,b2cf8cc7b17a3370] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:00:51.469 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:00:51.478 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:00:51.482 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:00:51.499 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:00:51.499 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:00:51.505 INFO 4486 --- [ocp-schedules-10,b9404d15959ffc80,b2cf8cc7b17a3370] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:00:51.906 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,3ed4e7cdaf9fc206,7a9935b473520667] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3ed4e7cdaf9fc206, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:51.906 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,3ed4e7cdaf9fc206,7a9935b473520667] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3ed4e7cdaf9fc206, duration=1 ms] 2024-10-16 14:00:52.213 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,7059c469e479c72a,8c433e7129301d70] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7059c469e479c72a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:52.214 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,7059c469e479c72a,8c433e7129301d70] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7059c469e479c72a, duration=1 ms] 2024-10-16 14:00:52.512 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:52.994 INFO 4486 --- [prometheus-minute-schedule-1,e5a39246c32fbb43,6dda3c63b414074f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:53.001 INFO 4486 --- [prometheus-minute-schedule-1,e5a39246c32fbb43,6dda3c63b414074f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:53.027 INFO 4486 --- [ocp-async-4,a8f1cabb8ee3771c,99bd3bd8d84d6995] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:00:53.377 INFO 4486 --- [ocp-async-5,67c8f46f1c226db0,b56037fbdaa036df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:55.378 INFO 4486 --- [ocp-async-2,36a4c3afd6144978,1202aab93eabe980] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:55.512 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:55.994 INFO 4486 --- [prometheus-minute-schedule-1,739223350e1ce24a,02207e9daf1eda12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:56.000 INFO 4486 --- [prometheus-minute-schedule-1,739223350e1ce24a,02207e9daf1eda12] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:57.378 INFO 4486 --- [ocp-async-8,e1386688cdca3410,516dce1ecd141d35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:00:58.357 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c709b4f7545757f6,b95b84f169dc8708] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c709b4f7545757f6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:00:58.358 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,c709b4f7545757f6,b95b84f169dc8708] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c709b4f7545757f6, duration=2 ms] 2024-10-16 14:00:58.514 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:00:58.994 INFO 4486 --- [prometheus-minute-schedule-1,1a4559754ac14441,f453e32f794ef652] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058460, interval=60 2024-10-16 14:00:59.000 INFO 4486 --- [prometheus-minute-schedule-1,1a4559754ac14441,f453e32f794ef652] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:00:59.256 INFO 4486 --- [ocp-schedules-10,dbc038b1521c1f64,867668229cf0c180] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:00:59.378 INFO 4486 --- [ocp-async-3,dbecd036fe6c27ab,cb2a12d7e72323ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:00.142 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,5600da66420ddd6b] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 13:01:00.0, nextRun=2024-10-16 14:01:00.0, currentDate=Wed Oct 16 14:01:00 CST 2024 2024-10-16 14:01:00.142 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,5600da66420ddd6b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999995, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=2024-10-16 13:01:00.0, nextRunTime=2024-10-16 14:01:00.0} 2024-10-16 14:01:00.144 WARN 4486 --- [task-coordinator3,791e68b70ade5cad,5600da66420ddd6b] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-10-16 14:01:00.144 WARN 4486 --- [task-coordinator3,791e68b70ade5cad,5600da66420ddd6b] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2024-10-16 14:01:00.145 WARN 4486 --- [task-coordinator3,791e68b70ade5cad,5600da66420ddd6b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : failed to get template for class, action=com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule java.lang.NullPointerException: null at com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule.ready(CollectClusterSnapshotSchedule.java:74) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.getTaskMeta(ReadyTaskWorker.java:366) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.checkAndScheduleCronTask(ReadyTaskWorker.java:173) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.lambda$loopScheduleTask$1(ReadyTaskWorker.java:134) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.task.engine.coordinator.worker.AbstractTaskWorker.transactionWithoutResult(AbstractTaskWorker.java:61) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.loopScheduleTask(ReadyTaskWorker.java:133) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.work(ReadyTaskWorker.java:102) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lockAndExecute(TaskCoordinator.java:137) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$startup$0(TaskCoordinator.java:75) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2024-10-16 14:01:00.147 INFO 4486 --- [task-coordinator3,791e68b70ade5cad,5600da66420ddd6b] c.o.o.c.t.e.c.worker.SubtaskGenerator : Split list is empty, name=Collect cluster hour metric, splitKey=cluster_ids, context=Argument{parallelIdx=-1, stringMap={}, listMap={}} 2024-10-16 14:01:00.157 WARN 4486 --- [task-coordinator3,791e68b70ade5cad,5600da66420ddd6b] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Subtask is empty, mark task failed, name=Collect cluster hour metric, taskId=5266498 2024-10-16 14:01:00.382 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,56a4d15309d8229e,1994d7f2a7f10811] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=56a4d15309d8229e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:00.382 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,56a4d15309d8229e,1994d7f2a7f10811] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=56a4d15309d8229e, duration=0 ms] 2024-10-16 14:01:01.128 INFO 4486 --- [ocp-schedules-12,7ecc758f558c0c65,f6e223d284e27e08] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:01:01.179 WARN 4486 --- [scanner-13318276-1,7ecc758f558c0c65,f3a54f69b85ec888] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:01:01.253 INFO 4486 --- [ocp-schedules-30,4dc7b9c0f8e5d6f4,4f6254cec76da2c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:01:01.268 INFO 4486 --- [alarm-detect-5,7ecc758f558c0c65,441e3c25d8f670d0] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058430 ~ 1729058451 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058445, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058445, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058445=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:01:01.380 INFO 4486 --- [ocp-async-5,fd99dd8e3c070cc6,f8c8f1d09d579d5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:01.515 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:01.909 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9dd3445dfc4eac02,3cb543380b72c5d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9dd3445dfc4eac02, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:01.910 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,9dd3445dfc4eac02,3cb543380b72c5d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9dd3445dfc4eac02, duration=1 ms] 2024-10-16 14:01:01.994 INFO 4486 --- [prometheus-minute-schedule-1,2dd20010909e09a4,2e497ef63fe517cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:02.000 INFO 4486 --- [prometheus-minute-schedule-1,2dd20010909e09a4,2e497ef63fe517cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:02.217 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d77857a42bae7e53,91b47358003d78eb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d77857a42bae7e53, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:02.217 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,d77857a42bae7e53,91b47358003d78eb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d77857a42bae7e53, duration=0 ms] 2024-10-16 14:01:02.651 INFO 4486 --- [ocp-schedules-20,e4b02d54f9919179,d5c6d5eb56448b58] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:01:02.651+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:01:02.657 INFO 4486 --- [ocp-schedules-20,e4b02d54f9919179,d5c6d5eb56448b58] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:01:02.667 INFO 4486 --- [ocp-schedules-20,e4b02d54f9919179,d5c6d5eb56448b58] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:01:02.678 INFO 4486 --- [ocp-schedules-20,e4b02d54f9919179,d5c6d5eb56448b58] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:01:03.027 INFO 4486 --- [ocp-async-1,9af51f67825aacc0,3fd3a35e0f8d0934] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:01:03.380 INFO 4486 --- [ocp-async-7,2dbeea9143cabf06,e47be3525f1592f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:03.384 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,43db57d161722b8d,254a478441349e1e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=43db57d161722b8d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:01:03.385 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,43db57d161722b8d,254a478441349e1e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=43db57d161722b8d, duration=2 ms] 2024-10-16 14:01:04.516 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:04.994 INFO 4486 --- [prometheus-minute-schedule-1,b81723df87f3deed,2ccf064c67eb29bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:05.000 INFO 4486 --- [prometheus-minute-schedule-1,b81723df87f3deed,2ccf064c67eb29bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:05.380 INFO 4486 --- [ocp-async-6,cf8a4b345faba392,647b95f946193fbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:07.066 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:01:07.066 INFO 4486 --- [ocp-schedules-23,2d9205646da0342c,8419ed62d8f86e4e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:01:07.069 INFO 4486 --- [ocp-schedules-23,2d9205646da0342c,8419ed62d8f86e4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:07.081 INFO 4486 --- [ocp-schedules-30,9634a5cec7972e6c,72e7e1447cdbd212] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:01:07.083 INFO 4486 --- [ocp-schedules-30,9634a5cec7972e6c,72e7e1447cdbd212] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:01:07.093 INFO 4486 --- [ocp-schedules-20,fdfe6c939e4f672a,d4e832a39dd73ae5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:01:07.094 INFO 4486 --- [host-check-agent-status1,fdfe6c939e4f672a,e2d3c8c42c37536f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:01:07.095 INFO 4486 --- [ocp-schedules-15,d15db8358a1174a4,45d44bd8a4230bff] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:01:07.095 INFO 4486 --- [host-check-agent-status2,fdfe6c939e4f672a,5f0c3357a3d6cdd8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:01:07.096 INFO 4486 --- [host-check-agent-status3,fdfe6c939e4f672a,028dfc463fffe8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:01:07.096 INFO 4486 --- [host-check-agent-status4,fdfe6c939e4f672a,d853250455f73e0d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:01:07.097 INFO 4486 --- [host-check-clockdiff1,d15db8358a1174a4,ff855bf051bf8ad1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:01:07.097 INFO 4486 --- [host-check-clockdiff2,d15db8358a1174a4,1650ac696b15f2c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:01:07.099 INFO 4486 --- [host-check-clockdiff4,d15db8358a1174a4,2f4afd038cc314d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:01:07.100 INFO 4486 --- [host-check-clockdiff3,d15db8358a1174a4,20d41adb32666911] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:01:07.102 INFO 4486 --- [ocp-schedules-2,e86feb4435439252,41f4f2ab502d1618] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:01:07.104 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:01:07.105 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:01:07.105 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:01:07.107 INFO 4486 --- [host-check-agent-status1,fdfe6c939e4f672a,e2d3c8c42c37536f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:07.107 INFO 4486 --- [host-check-agent-status4,fdfe6c939e4f672a,d853250455f73e0d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:01:07.116 INFO 4486 --- [host-check-agent-status2,fdfe6c939e4f672a,5f0c3357a3d6cdd8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:01:07.119 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:01:07.120 INFO 4486 --- [host-check-agent-status3,fdfe6c939e4f672a,028dfc463fffe8b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:01:07.136 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.141 INFO 4486 --- [host-check-agent-status2,fdfe6c939e4f672a,5f0c3357a3d6cdd8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:01:07.141 INFO 4486 --- [host-check-agent-status4,fdfe6c939e4f672a,d853250455f73e0d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:01:07.141 INFO 4486 --- [host-check-agent-status1,fdfe6c939e4f672a,e2d3c8c42c37536f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:07.141 INFO 4486 --- [host-check-agent-status3,fdfe6c939e4f672a,028dfc463fffe8b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:01:07.141 INFO 4486 --- [host-check-clockdiff1,d15db8358a1174a4,ff855bf051bf8ad1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058467 0 0 , error= 2024-10-16 14:01:07.141 INFO 4486 --- [host-check-clockdiff2,d15db8358a1174a4,1650ac696b15f2c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058467 4 4 , error= 2024-10-16 14:01:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:07.145 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:07.146 INFO 4486 --- [host-check-clockdiff4,d15db8358a1174a4,2f4afd038cc314d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058467 4 4 , error= 2024-10-16 14:01:07.146 INFO 4486 --- [host-check-clockdiff3,d15db8358a1174a4,20d41adb32666911] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058467 4 4 , error= 2024-10-16 14:01:07.150 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:01:07.151 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:01:07.151 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:07.152 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:01:07.152 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:01:07.154 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:01:07.156 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:01:07.157 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:01:07.175 INFO 4486 --- [host-check-agent-status1,fdfe6c939e4f672a,e2d3c8c42c37536f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:01:07.175 INFO 4486 --- [host-check-agent-status4,fdfe6c939e4f672a,d853250455f73e0d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:01:07.176 INFO 4486 --- [host-check-agent-status2,fdfe6c939e4f672a,5f0c3357a3d6cdd8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:01:07.176 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:01:07.176 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.177 INFO 4486 --- [host-check-agent-status3,fdfe6c939e4f672a,028dfc463fffe8b2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:01:07.177 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:07.178 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:01:07.180 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:01:07.183594328+08:00 2024-10-16 14:01:07.180 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:01:07.179965767+08:00 2024-10-16 14:01:07.180 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:01:07.180+08:00 2024-10-16 14:01:07.181 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:01:07.180+08:00 2024-10-16 14:01:07.182 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status2,fdfe6c939e4f672a,5f0c3357a3d6cdd8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status3,fdfe6c939e4f672a,028dfc463fffe8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status3,fdfe6c939e4f672a,028dfc463fffe8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status4,fdfe6c939e4f672a,d853250455f73e0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status3,fdfe6c939e4f672a,028dfc463fffe8b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status4,fdfe6c939e4f672a,d853250455f73e0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status2,fdfe6c939e4f672a,5f0c3357a3d6cdd8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status4,fdfe6c939e4f672a,d853250455f73e0d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status2,fdfe6c939e4f672a,5f0c3357a3d6cdd8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status1,fdfe6c939e4f672a,e2d3c8c42c37536f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status1,fdfe6c939e4f672a,e2d3c8c42c37536f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:01:07.182 INFO 4486 --- [host-check-agent-status1,fdfe6c939e4f672a,e2d3c8c42c37536f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:01:07.183 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:01:07.183 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:01:07.180+08:00 and status AVAILABLE 2024-10-16 14:01:07.184 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:01:07.188279723+08:00 2024-10-16 14:01:07.184 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:01:07.180+08:00 and status AVAILABLE 2024-10-16 14:01:07.184 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:01:07.185 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:01:07.184+08:00 2024-10-16 14:01:07.185 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:01:07.188979798+08:00 2024-10-16 14:01:07.186 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:01:07.185+08:00 2024-10-16 14:01:07.186 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:01:07.187 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:01:07.188 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:01:07.184+08:00 and status AVAILABLE 2024-10-16 14:01:07.189 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:01:07.185+08:00 and status AVAILABLE 2024-10-16 14:01:07.189 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.191 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:01:07.191 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:01:07.193 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:01:07.196 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:01:07.196 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:07.198 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:07.198 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:01:07.200 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:01:07.200 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.201 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:01:07.201 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:01:07.202 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:01:07.203 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:01:07.213 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.214 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:01:07.221 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:07.222 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:07.227 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:07.228 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.229 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:01:07.232 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:07.233 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.234 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:01:07.246 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:07.247 INFO 4486 --- [ocp-schedules-9,aaf24224d75699dc,4194573f28e81fed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:01:07.256 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:01:07.262 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:01:07.264 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:01:07.266 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:01:07.274 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:01:07.275 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:01:07.276 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:01:07.281 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:07.286 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:01:07.290 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:07.291 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:07.301 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:07.309 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:01:07.320 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:01:07.321 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:01:07.332 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:01:07.335 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:07.342 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:01:07.349 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:07.353 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:01:07.359 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:01:07.366 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:01:07.370 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:01:07.372 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:07.377 INFO 4486 --- [host-check-host-status3,e86feb4435439252,f1ffbd70375aa390] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:01:07.380 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:01:07.381 INFO 4486 --- [ocp-async-1,41553747559b50dc,b2d29f9a67373785] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:07.389 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:07.392 INFO 4486 --- [host-check-host-status4,e86feb4435439252,ac2794fa8583a377] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:01:07.408 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:01:07.419 INFO 4486 --- [host-check-host-status2,e86feb4435439252,b67821efe7f7135f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:01:07.442 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:01:07.455 INFO 4486 --- [host-check-host-status1,e86feb4435439252,22cffc161147469d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:01:07.516 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:07.994 INFO 4486 --- [prometheus-minute-schedule-1,a6abb4924d313865,ced3435b9c616e28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:08.003 INFO 4486 --- [prometheus-minute-schedule-1,a6abb4924d313865,ced3435b9c616e28] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:08.364 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e569231733892152,e741290bffedc3d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e569231733892152, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:08.364 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e569231733892152,e741290bffedc3d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e569231733892152, duration=1 ms] 2024-10-16 14:01:09.002 INFO 4486 --- [ocp-schedules-16,679c6ace7cee88b7,9b5bcd4229d825f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:01:09.002 INFO 4486 --- [ocp-schedules-16,120bd7efe0dc77ee,07188f4ad3817dcb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:01:09.004 INFO 4486 --- [ocp-schedules-16,120bd7efe0dc77ee,07188f4ad3817dcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:09.034 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.034 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:09.043 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.045 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:09.068 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.069 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.069 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:01:09.074 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.075 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:09.100 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.101 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.101 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:09.102 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:01:09.102 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.105 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:01:09.107 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:09.118 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.118 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:09.131 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.140 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.141 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.142 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:09.143 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:01:09.150 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:01:09.154 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:09.157 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:09.182 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:01:09.188 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.190 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.191 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:01:09.210 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:01:09.210 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:01:09.244 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.246 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.247 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:09.251 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.254 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:01:09.258 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:09.260 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:01:09.274 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.275 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.275 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:01:09.400 INFO 4486 --- [ocp-async-5,edaff4824262ff83,aa6774e46db56d6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:09.411 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:01:09.411 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:01:09.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:09.452 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.452 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:09.486 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.486 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:09.519 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.519 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:09.545 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.545 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:09.571 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:09.594 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.596 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.597 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:09.601 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:01:09.605 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:01:09.611 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:01:09.613 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:01:09.634 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:01:09.766 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:01:09.792 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:09.799 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:09.802 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:01:09.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:01:09.831 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:01:10.006 INFO 4486 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:01:10.006 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:01:10.026 INFO 4486 --- [Check obproxy1,11db76d580dff09c,543479b628d664dc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:01:10.026 INFO 4486 --- [Check obproxy1,11db76d580dff09c,543479b628d664dc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:10.027 INFO 4486 --- [Check obproxy2,ee1c57ffea01111f,bcfa3e1c71adc81a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:01:10.027 INFO 4486 --- [Check obproxy2,ee1c57ffea01111f,bcfa3e1c71adc81a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:10.031 INFO 4486 --- [Collect obproxy1,e78da4487d992de0,96f2c340c3bb378d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:01:10.031 INFO 4486 --- [Collect obproxy1,e78da4487d992de0,96f2c340c3bb378d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:10.032 INFO 4486 --- [Collect obproxy2,05d85e32ef922f4c,34a051b4c22284af] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:01:10.032 INFO 4486 --- [Collect obproxy2,05d85e32ef922f4c,34a051b4c22284af] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:10.050 INFO 4486 --- [Check obproxy2,ee1c57ffea01111f,bcfa3e1c71adc81a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:01:10.050 INFO 4486 --- [Check obproxy2,ee1c57ffea01111f,bcfa3e1c71adc81a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:01:10.053 INFO 4486 --- [Collect obproxy2,05d85e32ef922f4c,34a051b4c22284af] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:01:10.059 INFO 4486 --- [Collect obproxy2,05d85e32ef922f4c,34a051b4c22284af] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:01:10.064 INFO 4486 --- [Collect obproxy1,e78da4487d992de0,96f2c340c3bb378d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:01:10.064 INFO 4486 --- [Collect obproxy2,05d85e32ef922f4c,34a051b4c22284af] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:01:10.066 INFO 4486 --- [Check obproxy1,11db76d580dff09c,543479b628d664dc] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:01:10.067 INFO 4486 --- [Check obproxy1,11db76d580dff09c,543479b628d664dc] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:01:10.070 INFO 4486 --- [Collect obproxy1,e78da4487d992de0,96f2c340c3bb378d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:01:10.073 INFO 4486 --- [Collect obproxy1,e78da4487d992de0,96f2c340c3bb378d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:01:10.388 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,36bcbe04ac0690b9,d4b1bb249d7b31a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=36bcbe04ac0690b9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:10.388 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,36bcbe04ac0690b9,d4b1bb249d7b31a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=36bcbe04ac0690b9, duration=1 ms] 2024-10-16 14:01:10.518 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:10.979 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:01:10.994 INFO 4486 --- [prometheus-minute-schedule-1,acab843bf5f75c9f,db97946d824b3700] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:11.000 INFO 4486 --- [prometheus-minute-schedule-1,acab843bf5f75c9f,db97946d824b3700] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:11.008 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:01:11.272 INFO 4486 --- [ocp-schedules-11,7f0af81398050d56,f4a69f6a1aab0c14] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:01:11.401 INFO 4486 --- [ocp-async-7,ae91c6de203375fe,874e44f8f2c633ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:11.440 INFO 4486 --- [ocp-schedules-10,0bbc59c31e6b8e73,3d09c4587c78cf80] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:01:11.462 WARN 4486 --- [scanner-13318634-1,0bbc59c31e6b8e73,fc17be16d317e629] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:01:11.519 INFO 4486 --- [alarm-detect-20,0bbc59c31e6b8e73,276c2ab8fc5724c7] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058445 ~ 1729058461 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058460, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058460, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058460=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:01:11.915 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,759e2849143ebc1e,63b429fb982b9ceb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=759e2849143ebc1e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:11.916 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,759e2849143ebc1e,63b429fb982b9ceb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=759e2849143ebc1e, duration=2 ms] 2024-10-16 14:01:12.221 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,70a2f73ba8b86b2b,c96575dc1b7c0de9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=70a2f73ba8b86b2b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:12.221 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,70a2f73ba8b86b2b,c96575dc1b7c0de9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=70a2f73ba8b86b2b, duration=1 ms] 2024-10-16 14:01:13.027 INFO 4486 --- [ocp-async-8,31e68a9f69c0a0f0,33556ca108289047] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:01:13.401 INFO 4486 --- [ocp-async-6,55e74a8ce1af6ea4,a7ee45a69a3945d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:13.518 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:13.577 INFO 4486 --- [ocp-schedules-9,aa46012f7d04ecd9,64ed188d881e12c8] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:01:13.579 INFO 4486 --- [ocp-schedules-9,aa46012f7d04ecd9,64ed188d881e12c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:13.579 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:01:13.580 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:01:13.580 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:01:13.994 INFO 4486 --- [prometheus-minute-schedule-1,208af79509045ce1,3cfc4e5fc7353e89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:14.000 INFO 4486 --- [prometheus-minute-schedule-1,208af79509045ce1,3cfc4e5fc7353e89] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:14.533 ERROR 4486 --- [metric-archive-second-26,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=835727) Duplicate entry '586679390278586368-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.531922] [YB420AC531B8-00060B7DB4196BCC-0-0]; nested exception is java.sql.BatchUpdateException: (conn=835727) Duplicate entry '586679390278586368-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.531922] [YB420AC531B8-00060B7DB4196BCC-0-0] 2024-10-16 14:01:14.535 ERROR 4486 --- [metric-archive-second-25,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2949508) Duplicate entry '669741696482349056-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.534445] [YB420AC531B8-00060B7DB76886AB-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2949508) Duplicate entry '669741696482349056-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.534445] [YB420AC531B8-00060B7DB76886AB-0-0] 2024-10-16 14:01:14.535 ERROR 4486 --- [metric-archive-second-15,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '586679389997568000-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.534666] [YB420AC531B8-00060B7DC0C893F3-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '586679389997568000-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.534666] [YB420AC531B8-00060B7DC0C893F3-0-0] 2024-10-16 14:01:14.535 ERROR 4486 --- [metric-archive-second-17,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2457465) Duplicate entry '586774769053536256-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.534600] [YB420AC531B8-00060B7DAD2B86DA-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2457465) Duplicate entry '586774769053536256-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.534600] [YB420AC531B8-00060B7DAD2B86DA-0-0] 2024-10-16 14:01:14.537 ERROR 4486 --- [metric-archive-second-20,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1694585) Duplicate entry '586774769321971712-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.536401] [YB420AC531B8-00060B7DA1ABEF92-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1694585) Duplicate entry '586774769321971712-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.536401] [YB420AC531B8-00060B7DA1ABEF92-0-0] 2024-10-16 14:01:14.537 ERROR 4486 --- [metric-archive-second-32,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '669741696503320576-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.536790] [YB420AC531B8-00060B7D84EAB3BF-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '669741696503320576-1729058400' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:01:14.536790] [YB420AC531B8-00060B7D84EAB3BF-0-0] 2024-10-16 14:01:14.747 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058400, nodeCount=343, affectRows=343 2024-10-16 14:01:14.748 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058400, nodeCount=343, validCount=343 2024-10-16 14:01:14.832 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058400, nodeCount=496, affectRows=496 2024-10-16 14:01:14.834 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058400, nodeCount=496, validCount=496 2024-10-16 14:01:14.835 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058400, nodeCount=490, affectRows=490 2024-10-16 14:01:14.837 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058400, nodeCount=490, validCount=490 2024-10-16 14:01:14.874 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058400, nodeCount=603, affectRows=603 2024-10-16 14:01:14.876 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058400, nodeCount=601, validCount=601 2024-10-16 14:01:14.995 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058400, nodeCount=748, affectRows=748 2024-10-16 14:01:14.998 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058400, nodeCount=748, validCount=748 2024-10-16 14:01:15.015 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058400, nodeCount=769, affectRows=769 2024-10-16 14:01:15.018 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058400, nodeCount=768, validCount=768 2024-10-16 14:01:15.189 INFO 4486 --- [metric-archive-async-22,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058400, nodeCount=41, affectRows=41 2024-10-16 14:01:15.189 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058400, nodeCount=46, affectRows=46 2024-10-16 14:01:15.198 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058400, nodeCount=202, affectRows=202 2024-10-16 14:01:15.199 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058400, nodeCount=191, affectRows=191 2024-10-16 14:01:15.240 INFO 4486 --- [metric-archive-async-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058400, nodeCount=41, validCount=41 2024-10-16 14:01:15.243 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058400, nodeCount=46, validCount=46 2024-10-16 14:01:15.401 INFO 4486 --- [ocp-async-4,a7aeff7817ea4bf8,2647fe00e4127051] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:15.407 INFO 4486 --- [metric-archive-async-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058400, nodeCount=191, validCount=191 2024-10-16 14:01:15.425 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058400, nodeCount=202, validCount=202 2024-10-16 14:01:16.519 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:16.994 INFO 4486 --- [prometheus-minute-schedule-1,7550ebfbab762505,6c4b2f0cecb50c2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:17.002 INFO 4486 --- [prometheus-minute-schedule-1,7550ebfbab762505,6c4b2f0cecb50c2e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:17.071 INFO 4486 --- [ocp-schedules-23,2d9205646da0342c,8419ed62d8f86e4e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:01:17.182 INFO 4486 --- [ocp-schedules-30,9634a5cec7972e6c,72e7e1447cdbd212] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:01:17.278 INFO 4486 --- [ocp-schedules-11,6d8eab48526f12b1,a8dc2a037fbefb7f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:01:17.401 INFO 4486 --- [ocp-async-10,63066aad924ad093,a1859f126bb1a15a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:18.369 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,062b1d2ed29965e4,95d2eba4d2828492] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=062b1d2ed29965e4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:18.369 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,062b1d2ed29965e4,95d2eba4d2828492] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=062b1d2ed29965e4, duration=1 ms] 2024-10-16 14:01:19.262 INFO 4486 --- [ocp-schedules-26,be223eaca96361b3,6284fe2c9b2d791b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:01:19.402 INFO 4486 --- [ocp-async-9,ee2cbaf0283a712c,24198e2d8c1eb5d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:19.520 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:19.994 INFO 4486 --- [prometheus-minute-schedule-1,065f6647477337db,e7b44356865fea71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:20.001 INFO 4486 --- [prometheus-minute-schedule-1,065f6647477337db,e7b44356865fea71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:20.396 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9c8d5aeb19cbe5f5,fe4556eff845bced] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9c8d5aeb19cbe5f5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:20.396 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9c8d5aeb19cbe5f5,fe4556eff845bced] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9c8d5aeb19cbe5f5, duration=2 ms] 2024-10-16 14:01:20.996 INFO 4486 --- [prometheus-cache-monitor1,2f05dfb48dd4982c,8f77ff6fe24ba4f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13802,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":16597} 2024-10-16 14:01:20.996 INFO 4486 --- [prometheus-cache-monitor1,2f05dfb48dd4982c,8f77ff6fe24ba4f0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:01:21.277 INFO 4486 --- [ocp-schedules-8,2656ea6eadf20ffe,1cf652786ee3d557] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:01:21.402 INFO 4486 --- [ocp-async-2,c4bcd03118e64de0,31f826acc6cd06b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:21.568 INFO 4486 --- [ocp-schedules-7,10c37b8486984ab6,cab055499fcc278a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:01:21.597 WARN 4486 --- [scanner-13319659-1,10c37b8486984ab6,7a151bb3f2b1c731] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:01:21.921 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4014c2cb65bba589,59b3076c6deca61b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4014c2cb65bba589, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:21.921 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,4014c2cb65bba589,59b3076c6deca61b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4014c2cb65bba589, duration=1 ms] 2024-10-16 14:01:22.231 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2920a9f8b34fbadc,89742a5827d661f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2920a9f8b34fbadc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:22.232 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2920a9f8b34fbadc,89742a5827d661f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2920a9f8b34fbadc, duration=2 ms] 2024-10-16 14:01:22.527 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:22.994 INFO 4486 --- [prometheus-minute-schedule-1,31e5b3a043c5b5f4,9679137de75d250d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:23.000 INFO 4486 --- [prometheus-minute-schedule-1,31e5b3a043c5b5f4,9679137de75d250d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:23.027 INFO 4486 --- [ocp-async-1,294872c7d2eed260,0c5aad3d625dd3d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:01:23.402 INFO 4486 --- [ocp-async-8,0b08bc30082d9f68,061d5c811b1c894f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:23.580 INFO 4486 --- [ocp-schedules-9,aa46012f7d04ecd9,64ed188d881e12c8] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:01:23.926 INFO 4486 --- [ocp-schedules-30,e5cad4f43b0d9a38,1f3f43dbeca429a7] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:01:25.215 INFO 4486 --- [ocp-schedules-28,29d1ff897cdcc4fa,d168a7c9b3040ecf] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:01:25.402 INFO 4486 --- [ocp-async-9,56265968d57fa1e0,5c98731083d4230d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:25.528 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:25.994 INFO 4486 --- [prometheus-minute-schedule-1,16262e11ceb53a7d,318bf2d3a6273b44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:26.000 INFO 4486 --- [prometheus-minute-schedule-1,16262e11ceb53a7d,318bf2d3a6273b44] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:26.767 INFO 4486 --- [ocp-schedules-21,4de08d8eb016593e,a2cb9b8a5a83cb74] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:01:26.767 INFO 4486 --- [ocp-schedules-21,4de08d8eb016593e,a2cb9b8a5a83cb74] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:01:26.807 INFO 4486 --- [Collect all tenants major compactions2,4de08d8eb016593e,4e35cb2c0f2ed05f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:01:26.807 INFO 4486 --- [Collect all tenants major compactions3,4de08d8eb016593e,2bd5fd31b0f9a06e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:01:26.807 INFO 4486 --- [Collect all tenants major compactions4,4de08d8eb016593e,e132b8eb0a1e8dd4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:01:26.807 INFO 4486 --- [Collect all tenants major compactions1,4de08d8eb016593e,dd66426224ea0b57] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:01:26.808 INFO 4486 --- [Collect all tenants major compactions5,4de08d8eb016593e,1bbc8f28db5ebff2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:01:26.821 INFO 4486 --- [Collect all tenants major compactions2,4de08d8eb016593e,4e35cb2c0f2ed05f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:26.822 INFO 4486 --- [Collect all tenants major compactions1,4de08d8eb016593e,dd66426224ea0b57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:26.822 INFO 4486 --- [Collect all tenants major compactions2,4de08d8eb016593e,4e35cb2c0f2ed05f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:01:26.823 INFO 4486 --- [Collect all tenants major compactions1,4de08d8eb016593e,dd66426224ea0b57] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:01:26.827 INFO 4486 --- [Collect all tenants major compactions2,4de08d8eb016593e,4e35cb2c0f2ed05f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:01:26.827 INFO 4486 --- [Collect all tenants major compactions1,4de08d8eb016593e,dd66426224ea0b57] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:01:26.828 INFO 4486 --- [Collect all tenants major compactions2,4de08d8eb016593e,4e35cb2c0f2ed05f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:01:26.828 INFO 4486 --- [Collect all tenants major compactions1,4de08d8eb016593e,dd66426224ea0b57] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:01:26.830 INFO 4486 --- [Collect all tenants major compactions3,4de08d8eb016593e,2bd5fd31b0f9a06e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:26.832 INFO 4486 --- [Collect all tenants major compactions3,4de08d8eb016593e,2bd5fd31b0f9a06e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:01:26.837 INFO 4486 --- [Collect all tenants major compactions3,4de08d8eb016593e,2bd5fd31b0f9a06e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:01:26.838 INFO 4486 --- [Collect all tenants major compactions3,4de08d8eb016593e,2bd5fd31b0f9a06e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:01:26.843 INFO 4486 --- [Collect all tenants major compactions5,4de08d8eb016593e,1bbc8f28db5ebff2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:26.844 INFO 4486 --- [Collect all tenants major compactions4,4de08d8eb016593e,e132b8eb0a1e8dd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:26.845 INFO 4486 --- [Collect all tenants major compactions5,4de08d8eb016593e,1bbc8f28db5ebff2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:01:26.847 INFO 4486 --- [Collect all tenants major compactions4,4de08d8eb016593e,e132b8eb0a1e8dd4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:01:26.851 INFO 4486 --- [Collect all tenants major compactions4,4de08d8eb016593e,e132b8eb0a1e8dd4] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:01:26.851 INFO 4486 --- [Collect all tenants major compactions5,4de08d8eb016593e,1bbc8f28db5ebff2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:01:26.851 INFO 4486 --- [Collect all tenants major compactions5,4de08d8eb016593e,1bbc8f28db5ebff2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:01:26.851 INFO 4486 --- [Collect all tenants major compactions4,4de08d8eb016593e,e132b8eb0a1e8dd4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:01:26.852 INFO 4486 --- [ocp-schedules-21,4de08d8eb016593e,a2cb9b8a5a83cb74] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:01:27.402 INFO 4486 --- [ocp-async-1,01744b0de1c58568,e0412eb40ce995ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:27.864 INFO 4486 --- [ocp-schedules-8,2e38fb87e6f81236,652db50112e29cf2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:01:27.863+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:01:27.865 INFO 4486 --- [ocp-schedules-8,2e38fb87e6f81236,652db50112e29cf2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:01:27.866 INFO 4486 --- [ocp-schedules-8,2e38fb87e6f81236,652db50112e29cf2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:01:27.867 INFO 4486 --- [ocp-schedules-8,2e38fb87e6f81236,652db50112e29cf2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:01:28.375 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,3e4b3bea360671f0,7a8c6221f47de765] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3e4b3bea360671f0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:28.375 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,3e4b3bea360671f0,7a8c6221f47de765] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3e4b3bea360671f0, duration=2 ms] 2024-10-16 14:01:28.530 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:28.826 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,07326d3b18f7c888,68b3ad63f0877f72] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=07326d3b18f7c888, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:01:28.827 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,07326d3b18f7c888,68b3ad63f0877f72] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=07326d3b18f7c888, duration=2 ms] 2024-10-16 14:01:28.994 INFO 4486 --- [prometheus-minute-schedule-1,f8fce4561f3e53b3,2a9072701ebe4538] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:29.000 INFO 4486 --- [prometheus-minute-schedule-1,f8fce4561f3e53b3,2a9072701ebe4538] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:29.010 INFO 4486 --- [ocp-schedules-20,67dcbf4504472cb8,7ecbf25a5a4ea685] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 14:01:29.012 INFO 4486 --- [ocp-schedules-20,67dcbf4504472cb8,7ecbf25a5a4ea685] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:29.161 INFO 4486 --- [ocp-schedules-5,5213cc12084fb15c,901d0fff9693e11b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 14:01:29.403 INFO 4486 --- [ocp-async-10,cf03669debb8453c,8bb9798baa88cd93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:30.103 INFO 4486 --- [prometheus-second-schedule-1,f998f4eda906e08e,2121ce9719f92236] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:01:30.400 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,02f0c863b49e1b4c,4552f4202bc28212] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=02f0c863b49e1b4c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:30.400 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,02f0c863b49e1b4c,4552f4202bc28212] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=02f0c863b49e1b4c, duration=1 ms] 2024-10-16 14:01:31.283 INFO 4486 --- [ocp-schedules-7,157071c91fae8474,36ad0ceee8487ff4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:01:31.403 INFO 4486 --- [ocp-async-3,00ba51fa44356875,0745e5660606c8c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:31.531 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:31.801 INFO 4486 --- [ocp-schedules-28,14d6c2d408666616,c33d5004aaaed879] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:01:31.870 WARN 4486 --- [scanner-13320039-1,14d6c2d408666616,72a3c2d00434cc38] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:01:31.882 INFO 4486 --- [alarm-detect-7,14d6c2d408666616,5e2b5ffdac4f3bff] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058460 ~ 1729058481 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058475, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058475, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058475=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:01:31.934 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f9ff8318088fd487,dd7be38e9c5280ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f9ff8318088fd487, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:31.935 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,f9ff8318088fd487,dd7be38e9c5280ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f9ff8318088fd487, duration=10 ms] 2024-10-16 14:01:31.994 INFO 4486 --- [prometheus-minute-schedule-1,c915146b143f3004,9d9fdc84583ed9ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:32.002 INFO 4486 --- [prometheus-minute-schedule-1,c915146b143f3004,9d9fdc84583ed9ba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:32.236 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1bba2286f5f533b6,5a0ddd48ac1612ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1bba2286f5f533b6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:32.236 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,1bba2286f5f533b6,5a0ddd48ac1612ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1bba2286f5f533b6, duration=1 ms] 2024-10-16 14:01:33.028 INFO 4486 --- [ocp-async-8,9cfca219cee3b464,d6c18f832dde363e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:01:33.403 INFO 4486 --- [ocp-async-2,98e527215e96392a,ed302d21f0abffaa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:34.532 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:34.994 INFO 4486 --- [prometheus-minute-schedule-1,a0ae1c0ace1ada3e,92f1d7756a495518] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:35.000 INFO 4486 --- [prometheus-minute-schedule-1,a0ae1c0ace1ada3e,92f1d7756a495518] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:35.403 INFO 4486 --- [ocp-async-4,84fe64b37f33318a,8f7a34ca411c6825] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:37.063 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:01:37.064 INFO 4486 --- [ocp-schedules-1,8442f2b6d3afef06,afdf9d3a9896ee5f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:01:37.066 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:01:37.069 INFO 4486 --- [ocp-schedules-1,8442f2b6d3afef06,afdf9d3a9896ee5f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:01:37.070 INFO 4486 --- [ocp-schedules-1,28ed5fe509ca9eba,fede4cfe566d8a2d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:01:37.070 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:01:37.071 INFO 4486 --- [ocp-schedules-1,28ed5fe509ca9eba,fede4cfe566d8a2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:37.081 INFO 4486 --- [ocp-schedules-26,7701d201749d6bd3,124f4c10c2276ac0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:01:37.083 INFO 4486 --- [ocp-schedules-26,7701d201749d6bd3,124f4c10c2276ac0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:37.083 INFO 4486 --- [ocp-schedules-18,fd886e3898b2d71b,571a5241782f18d4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 14:01:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:01:37.085 INFO 4486 --- [ocp-schedules-18,fd886e3898b2d71b,571a5241782f18d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:37.089 INFO 4486 --- [ocp-schedules-15,5b53f03506272e81,62c4ca71ca112fb6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:01:37.090 INFO 4486 --- [ocp-schedules-4,88e4d7fefcabe925,5c2bd7792f6c2f3f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:01:37.090 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:01:37.091 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:01:37.091 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:01:37.091 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:01:37.092 INFO 4486 --- [host-check-clockdiff1,88e4d7fefcabe925,67fe213d8781f1b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:01:37.094 INFO 4486 --- [ocp-schedules-17,cf7a903699431c6a,11810385238ec6af] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:01:37.095 INFO 4486 --- [host-check-clockdiff2,88e4d7fefcabe925,91fd418bb8c75892] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:01:37.097 INFO 4486 --- [host-check-agent-status1,cf7a903699431c6a,34a96bdfaa08162b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:01:37.099 INFO 4486 --- [host-check-clockdiff3,88e4d7fefcabe925,ea5865b749e58b09] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:01:37.099 INFO 4486 --- [host-check-agent-status2,cf7a903699431c6a,402a9f5588e95eb1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:01:37.099 INFO 4486 --- [host-check-agent-status3,cf7a903699431c6a,4d9db38f5325f4d0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:01:37.100 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:37.100 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:01:37.100 INFO 4486 --- [host-check-agent-status4,cf7a903699431c6a,4de9f280e0de0732] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:01:37.100 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:01:37.101 INFO 4486 --- [host-check-clockdiff4,88e4d7fefcabe925,5c54b82c20285d82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:01:37.103 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:37.103 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:01:37.104 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:01:37.105 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:01:37.106 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:01:37.106 INFO 4486 --- [host-check-agent-status1,cf7a903699431c6a,34a96bdfaa08162b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:37.107 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:37.107 INFO 4486 --- [host-check-agent-status3,cf7a903699431c6a,4d9db38f5325f4d0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:01:37.128 INFO 4486 --- [host-check-agent-status1,cf7a903699431c6a,34a96bdfaa08162b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:37.159 INFO 4486 --- [host-check-clockdiff3,88e4d7fefcabe925,ea5865b749e58b09] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058497 4 4 , error= 2024-10-16 14:01:37.159 INFO 4486 --- [host-check-agent-status2,cf7a903699431c6a,402a9f5588e95eb1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:01:37.160 INFO 4486 --- [host-check-clockdiff1,88e4d7fefcabe925,67fe213d8781f1b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058497 0 0 , error= 2024-10-16 14:01:37.160 INFO 4486 --- [host-check-agent-status3,cf7a903699431c6a,4d9db38f5325f4d0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:01:37.165 INFO 4486 --- [host-check-clockdiff2,88e4d7fefcabe925,91fd418bb8c75892] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058497 4 4 , error= 2024-10-16 14:01:37.170 INFO 4486 --- [host-check-agent-status4,cf7a903699431c6a,4de9f280e0de0732] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:01:37.170 INFO 4486 --- [host-check-agent-status2,cf7a903699431c6a,402a9f5588e95eb1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:01:37.171 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.172 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:37.173 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:01:37.173 INFO 4486 --- [host-check-agent-status4,cf7a903699431c6a,4de9f280e0de0732] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:01:37.180 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.181 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 14:01:37.194 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:37.195 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8d9618597845c796,c908178ac1dea9d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=8d9618597845c796, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 14:01:37.196 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8d9618597845c796,c908178ac1dea9d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=8d9618597845c796, duration=2 ms] 2024-10-16 14:01:37.197 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.199 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.199 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 14:01:37.201 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:01:37.202 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:01:37.202 INFO 4486 --- [host-check-agent-status3,cf7a903699431c6a,4d9db38f5325f4d0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:01:37.202 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.203 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:01:37.203 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.203 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 14:01:37.204 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:37.204 INFO 4486 --- [host-check-agent-status2,cf7a903699431c6a,402a9f5588e95eb1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:01:37.205 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:01:37.206 INFO 4486 --- [host-check-agent-status1,cf7a903699431c6a,34a96bdfaa08162b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:01:37.206 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:01:37.206 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:01:37.205917398+08:00 2024-10-16 14:01:37.206 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:01:37.210049221+08:00 2024-10-16 14:01:37.206 INFO 4486 --- [host-check-agent-status4,cf7a903699431c6a,4de9f280e0de0732] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:01:37.207 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:01:37.206+08:00 2024-10-16 14:01:37.207 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:01:37.206+08:00 2024-10-16 14:01:37.208 INFO 4486 --- [host-check-agent-status3,cf7a903699431c6a,4d9db38f5325f4d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:01:37.208 INFO 4486 --- [host-check-agent-status3,cf7a903699431c6a,4d9db38f5325f4d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:01:37.208 INFO 4486 --- [host-check-agent-status3,cf7a903699431c6a,4d9db38f5325f4d0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:01:37.208 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:01:37.211889325+08:00 2024-10-16 14:01:37.208 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:01:37.212223461+08:00 2024-10-16 14:01:37.209 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:01:37.208+08:00 2024-10-16 14:01:37.209 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.209 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:01:37.208+08:00 2024-10-16 14:01:37.209 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.210 INFO 4486 --- [host-check-agent-status2,cf7a903699431c6a,402a9f5588e95eb1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:01:37.210 INFO 4486 --- [host-check-agent-status2,cf7a903699431c6a,402a9f5588e95eb1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:01:37.210 INFO 4486 --- [host-check-agent-status2,cf7a903699431c6a,402a9f5588e95eb1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:01:37.210 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:01:37.206+08:00 and status AVAILABLE 2024-10-16 14:01:37.210 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:01:37.206+08:00 and status AVAILABLE 2024-10-16 14:01:37.210 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:37.211 INFO 4486 --- [host-check-agent-status1,cf7a903699431c6a,34a96bdfaa08162b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:01:37.211 INFO 4486 --- [host-check-agent-status1,cf7a903699431c6a,34a96bdfaa08162b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:01:37.211 INFO 4486 --- [host-check-agent-status1,cf7a903699431c6a,34a96bdfaa08162b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:01:37.212 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:01:37.208+08:00 and status AVAILABLE 2024-10-16 14:01:37.212 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:01:37.212 INFO 4486 --- [host-check-agent-status4,cf7a903699431c6a,4de9f280e0de0732] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:01:37.212 INFO 4486 --- [host-check-agent-status4,cf7a903699431c6a,4de9f280e0de0732] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:01:37.212 INFO 4486 --- [host-check-agent-status4,cf7a903699431c6a,4de9f280e0de0732] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:01:37.213 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:01:37.208+08:00 and status AVAILABLE 2024-10-16 14:01:37.213 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 14:01:37.213 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.214 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:01:37.214 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:01:37.214 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:01:37.214 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:01:37.215 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:01:37.215 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.216 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.217 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:01:37.217 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:01:37.221 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.222 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 14:01:37.223 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:37.223 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.223 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:01:37.224 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:01:37.225 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:01:37.225 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:37.225 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:01:37.227 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:01:37.227 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:01:37.229 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:01:37.230 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.230 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.230 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.230 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.230 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.231 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.234 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.234 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.235 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:01:37.237 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 14:01:37.238 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.239 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:01:37.241 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.246 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 14:01:37.250 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.251 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:37.252 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:37.262 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:37.262 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.263 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 14:01:37.263 INFO 4486 --- [ocp-schedules-6,5d227e0db93faacc,6441e3f6abd5594b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:01:37.271 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:01:37.272 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:01:37.289 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:01:37.298 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:01:37.300 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:01:37.309 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:01:37.313 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:01:37.316 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:01:37.319 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:37.324 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:01:37.326 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:37.328 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:01:37.342 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:37.344 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:01:37.348 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:01:37.360 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:01:37.361 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:01:37.374 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:01:37.382 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:37.392 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:37.404 INFO 4486 --- [ocp-async-9,d7901401a4e15a57,889657ae102dec1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:37.405 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:01:37.408 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:01:37.413 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:01:37.417 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:01:37.418 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:01:37.422 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:01:37.423 INFO 4486 --- [host-check-host-status3,5b53f03506272e81,d864c30c980aff26] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:01:37.433 INFO 4486 --- [host-check-host-status4,5b53f03506272e81,ba8ac286256349a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:01:37.434 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:37.437 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:01:37.488 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:01:37.488 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:01:37.491 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:37.493 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.494 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:01:37.495 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.495 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.496 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:01:37.498 INFO 4486 --- [host-check-host-status2,5b53f03506272e81,fb386d397e95762e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:01:37.499 INFO 4486 --- [host-check-host-status1,5b53f03506272e81,30dcad1efaac8506] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:01:37.507 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.507 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.507 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.507 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.507 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.507 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.532 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:37.537 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:01:37.582 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:01:37.633 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:37.651 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:37.653 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.654 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:01:37.655 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:01:37.656 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.657 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:01:37.658 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.659 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.660 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:01:37.671 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.671 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.671 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.671 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.671 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.671 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.679 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:01:37.869 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:01:37.916 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:37.919 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.919 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:01:37.920 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.921 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.922 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:01:37.933 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.933 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.933 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.933 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.933 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:37.933 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:37.951 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:01:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:01:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:01:37.994 INFO 4486 --- [prometheus-minute-schedule-1,179c9595a28217ab,f80a07ca8f01b219] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:37.997 INFO 4486 --- [ocp-schedules-23,4c082ee4db756e96,91972b5e22f0ad52] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 14:01:37.998 INFO 4486 --- [ocp-schedules-23,4c082ee4db756e96,91972b5e22f0ad52] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 14:01:38.004 INFO 4486 --- [prometheus-minute-schedule-1,179c9595a28217ab,f80a07ca8f01b219] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:38.034 INFO 4486 --- [pool-10-thread-2,9ecaa5434240ad20,24a17aea0cfd7630] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 14:01:38.034 INFO 4486 --- [pool-10-thread-2,9ecaa5434240ad20,24a17aea0cfd7630] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 14:01:38.035 INFO 4486 --- [pool-10-thread-3,9ecaa5434240ad20,0ec36de78059ca75] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 14:01:38.036 INFO 4486 --- [pool-10-thread-4,9ecaa5434240ad20,51f5c72888d53b2c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 14:01:38.037 INFO 4486 --- [pool-10-thread-1,9ecaa5434240ad20,79dee991f81e8506] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 14:01:38.041 INFO 4486 --- [pool-overview-container-7,dc1bfeca0cf614b6,2477df481e09bbe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.042 INFO 4486 --- [ForkJoinPool-6-worker-8,9ecaa5434240ad20,1db0e3ad74a02f89] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 14:01:38.042 INFO 4486 --- [ForkJoinPool-6-worker-8,9ecaa5434240ad20,1db0e3ad74a02f89] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 14:01:38.042 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 14:01:38.042 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 14:01:38.043 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 14:01:38.043 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 14:01:38.043 INFO 4486 --- [pool-overview-container-7,dc1bfeca0cf614b6,2477df481e09bbe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 14:01:38.044 INFO 4486 --- [pool-10-thread-1,9ecaa5434240ad20,79dee991f81e8506] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 14:01:38.044 INFO 4486 --- [pool-10-thread-1,9ecaa5434240ad20,79dee991f81e8506] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 14:01:38.050 INFO 4486 --- [pool-10-thread-2,9ecaa5434240ad20,24a17aea0cfd7630] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 14:01:38.060 INFO 4486 --- [pool-10-thread-2,9ecaa5434240ad20,24a17aea0cfd7630] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 14:01:38.060 INFO 4486 --- [host-check-clockdiff4,88e4d7fefcabe925,5c54b82c20285d82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058498 4 4 , error= 2024-10-16 14:01:38.076 INFO 4486 --- [ForkJoinPool-6-worker-8,9ecaa5434240ad20,1db0e3ad74a02f89] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.076 INFO 4486 --- [ForkJoinPool-6-worker-8,9ecaa5434240ad20,1db0e3ad74a02f89] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:01:38.078 INFO 4486 --- [ForkJoinPool-6-worker-8,9ecaa5434240ad20,1db0e3ad74a02f89] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.078 INFO 4486 --- [ForkJoinPool-6-worker-8,9ecaa5434240ad20,1db0e3ad74a02f89] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:01:38.083 INFO 4486 --- [ForkJoinPool-6-worker-13,9ecaa5434240ad20,f82d5c5722109e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.085 INFO 4486 --- [pool-overview-container-7,dc1bfeca0cf614b6,2477df481e09bbe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.085 INFO 4486 --- [ForkJoinPool-6-worker-13,9ecaa5434240ad20,f82d5c5722109e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:01:38.085 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.087 INFO 4486 --- [pool-overview-container-7,dc1bfeca0cf614b6,2477df481e09bbe9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 14:01:38.088 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:01:38.091 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.095 WARN 4486 --- [ForkJoinPool-6-worker-8,9ecaa5434240ad20,1db0e3ad74a02f89] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 14:01:38.096 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.096 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.097 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:01:38.097 INFO 4486 --- [ForkJoinPool-6-worker-13,9ecaa5434240ad20,f82d5c5722109e2b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 14:01:38.098 INFO 4486 --- [ForkJoinPool-6-worker-13,9ecaa5434240ad20,f82d5c5722109e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.099 INFO 4486 --- [ForkJoinPool-6-worker-13,9ecaa5434240ad20,f82d5c5722109e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:01:38.101 INFO 4486 --- [ForkJoinPool-6-worker-13,9ecaa5434240ad20,f82d5c5722109e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.101 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:01:38.101 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:01:38.102 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.102 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.102 INFO 4486 --- [ForkJoinPool-6-worker-13,9ecaa5434240ad20,f82d5c5722109e2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:01:38.102 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:01:38.107 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 14:01:38.111 INFO 4486 --- [ForkJoinPool-6-worker-6,9ecaa5434240ad20,1f030321db164a24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.113 INFO 4486 --- [ForkJoinPool-6-worker-6,9ecaa5434240ad20,1f030321db164a24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:01:38.119 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 14:01:38.121 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.123 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:01:38.130 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.132 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:01:38.135 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:38.140 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.140 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:01:38.141 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.145 INFO 4486 --- [ForkJoinPool-6-worker-6,9ecaa5434240ad20,1f030321db164a24] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 14:01:38.146 INFO 4486 --- [ForkJoinPool-6-worker-6,9ecaa5434240ad20,1f030321db164a24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.147 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.147 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 14:01:38.148 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.148 INFO 4486 --- [ForkJoinPool-6-worker-6,9ecaa5434240ad20,1f030321db164a24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:01:38.149 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:01:38.151 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:01:38.153 INFO 4486 --- [ForkJoinPool-6-worker-6,9ecaa5434240ad20,1f030321db164a24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.153 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.154 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:01:38.155 INFO 4486 --- [ForkJoinPool-6-worker-6,9ecaa5434240ad20,1f030321db164a24] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:01:38.165 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.167 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:01:38.167 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.173 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 14:01:38.185 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 14:01:38.186 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.187 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.188 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.188 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:01:38.189 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:01:38.191 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.191 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 14:01:38.192 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:01:38.197 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.198 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:01:38.201 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.202 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:01:38.205 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 14:01:38.207 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.208 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:01:38.211 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:01:38.212 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 14:01:38.216 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.217 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.219 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:01:38.224 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 14:01:38.225 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 14:01:38.226 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.231 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 14:01:38.238 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.239 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 14:01:38.240 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:01:38, 2024-10-16 14:01:38] 2024-10-16 14:01:38.240 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.242 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:01:38.244 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.244 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:38.246 INFO 4486 --- [ForkJoinPool-6-worker-4,9ecaa5434240ad20,f7bb490d3a109dd3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:01:38.246 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.248 INFO 4486 --- [pool-10-thread-3,9ecaa5434240ad20,0ec36de78059ca75] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 14:01:38.249 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 14:01:38.251 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.252 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 14:01:38.253 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:01:38, 2024-10-16 14:01:38] 2024-10-16 14:01:38.260 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 14:01:38.262 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.264 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:01:38, 2024-10-16 14:01:38] 2024-10-16 14:01:38.267 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:38.268 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.274 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:38.274 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 14:01:38.275 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.276 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:01:38.277 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:38.278 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.279 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:01:38.281 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:38.281 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.282 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 14:01:38.284 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.285 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 14:01:38.287 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:01:38, 2024-10-16 14:01:38] 2024-10-16 14:01:38.288 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.288 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:38.288 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:38.288 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:38.288 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:38.288 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:01:38.288 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 14:01:38.288 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.289 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 14:01:38.289 INFO 4486 --- [ForkJoinPool-6-worker-15,9ecaa5434240ad20,e30d45b2433123e1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 14:01:38.290 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:01:38, 2024-10-16 14:01:38] 2024-10-16 14:01:38.319 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 14:01:38.321 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.323 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:01:38, 2024-10-16 14:01:38] 2024-10-16 14:01:38.326 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 14:01:38.326 INFO 4486 --- [ForkJoinPool-6-worker-1,9ecaa5434240ad20,82e32cf03c6d66e6] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 14:01:38.329 INFO 4486 --- [pool-10-thread-4,9ecaa5434240ad20,51f5c72888d53b2c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 14:01:38.331 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:38.332 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:38.338 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 14:01:38.352 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:01:38.382 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ad33214375af39b0,da04ba618be4253b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ad33214375af39b0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:38.382 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ad33214375af39b0,da04ba618be4253b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ad33214375af39b0, duration=2 ms] 2024-10-16 14:01:38.480 INFO 4486 --- [ocp-schedules-11,00777b8c15382131,50dc47e22f3901c7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13802,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":16597} 2024-10-16 14:01:38.480 INFO 4486 --- [ocp-schedules-11,00777b8c15382131,50dc47e22f3901c7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:01:38.565 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:01:38.359777, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:01:38.565 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:01:38.359777, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:01:38.603 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:01:38.359777} } 2024-10-16 14:01:38.659 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:01:38.668 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:01:38.671 INFO 4486 --- [ocp-schedules-9,f9d10dae4761f65b,d6af4ba1a9659e47] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:01:39.041 INFO 4486 --- [ocp-schedules-20,67dcbf4504472cb8,7ecbf25a5a4ea685] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 14:01:39.267 INFO 4486 --- [ocp-schedules-25,9c8c2eb8e22762b6,fa213b004e0b2c0c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:01:39.404 INFO 4486 --- [ocp-async-1,803fcba94030fb43,e59042b3aa334054] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:39.831 INFO 4486 --- [ocp-schedules-16,120bd7efe0dc77ee,07188f4ad3817dcb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:01:39.832 INFO 4486 --- [ocp-schedules-16,ef6b8cbd57588c69,d55ba5601b23c7d3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:01:39.834 INFO 4486 --- [ocp-schedules-16,ef6b8cbd57588c69,d55ba5601b23c7d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:01:39.858 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.859 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:39.869 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.870 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:39.887 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.888 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:39.889 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:39.890 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.890 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:01:39.920 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:39.921 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.922 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:39.922 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:01:39.923 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.926 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:01:39.927 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:39.936 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:39.936 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:39.947 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:39.955 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:39.956 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.956 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:39.957 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:01:39.962 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:01:39.966 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:39.968 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:39.984 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:39.985 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:39.986 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:01:39.989 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:01:40.006 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:01:40.034 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:01:40.050 INFO 4486 --- [Collect obproxy1,585d592b01aa95e1,68a4bd12fa3ebbd0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:01:40.050 INFO 4486 --- [Collect obproxy1,585d592b01aa95e1,68a4bd12fa3ebbd0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:40.051 INFO 4486 --- [Collect obproxy2,5a360494266cabf0,60618e839134316c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:01:40.051 INFO 4486 --- [Collect obproxy2,5a360494266cabf0,60618e839134316c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:40.051 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:40.053 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:40.053 INFO 4486 --- [Check obproxy1,52fcf6d97949422e,b21a69ff944ad7fa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:01:40.053 INFO 4486 --- [Check obproxy1,52fcf6d97949422e,b21a69ff944ad7fa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:40.053 INFO 4486 --- [Check obproxy2,917c23c9579a3754,4b8bb2b42066dc1e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:01:40.053 INFO 4486 --- [Check obproxy2,917c23c9579a3754,4b8bb2b42066dc1e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:01:40.054 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:01:40.054 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:01:40.054 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:01:40.057 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:40.060 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:01:40.063 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:40.067 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:01:40.115 INFO 4486 --- [Collect obproxy2,5a360494266cabf0,60618e839134316c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:01:40.117 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:40.117 INFO 4486 --- [Check obproxy2,917c23c9579a3754,4b8bb2b42066dc1e] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:01:40.117 INFO 4486 --- [Check obproxy2,917c23c9579a3754,4b8bb2b42066dc1e] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:01:40.117 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:40.118 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:01:40.120 INFO 4486 --- [Collect obproxy2,5a360494266cabf0,60618e839134316c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:01:40.124 INFO 4486 --- [Collect obproxy1,585d592b01aa95e1,68a4bd12fa3ebbd0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:01:40.125 INFO 4486 --- [Collect obproxy2,5a360494266cabf0,60618e839134316c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:01:40.125 INFO 4486 --- [Check obproxy1,52fcf6d97949422e,b21a69ff944ad7fa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:01:40.125 INFO 4486 --- [Check obproxy1,52fcf6d97949422e,b21a69ff944ad7fa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:01:40.127 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:01:40.127 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:01:40.129 INFO 4486 --- [Collect obproxy1,585d592b01aa95e1,68a4bd12fa3ebbd0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:01:40.129 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:40.129 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:40.134 INFO 4486 --- [Collect obproxy1,585d592b01aa95e1,68a4bd12fa3ebbd0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:01:40.155 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:40.155 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:40.179 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:40.179 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:40.203 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:40.204 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:01:40.229 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:40.229 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:01:40.255 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:01:40.279 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:40.281 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:40.282 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:01:40.286 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:01:40.291 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:01:40.295 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:01:40.297 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:01:40.319 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:01:40.406 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,39f8e6b95ee3525f,246d108514472ab5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=39f8e6b95ee3525f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:40.406 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,39f8e6b95ee3525f,246d108514472ab5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=39f8e6b95ee3525f, duration=1 ms] 2024-10-16 14:01:40.453 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:01:40.481 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:01:40.482 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:01:40.484 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:01:40.510 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:01:40.510 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:01:40.533 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:40.994 INFO 4486 --- [prometheus-minute-schedule-1,9f7858a4b72d68ed,a22f28156e8177de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:40.995 INFO 4486 --- [prometheus-hour-schedule-1,d1962595fa837aa4,7a7ce5fc3d52d8c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:01:41.000 INFO 4486 --- [prometheus-minute-schedule-1,9f7858a4b72d68ed,a22f28156e8177de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:41.288 INFO 4486 --- [ocp-schedules-25,c6876a58d4ad30da,eab853452710ec16] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:01:41.404 INFO 4486 --- [ocp-async-4,a778c9fe5f779789,10cb6e3fa3705c46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:41.941 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5630a851ff8bf081,ea9a2efe5b2f81f9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5630a851ff8bf081, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:41.941 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5630a851ff8bf081,ea9a2efe5b2f81f9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5630a851ff8bf081, duration=1 ms] 2024-10-16 14:01:41.983 INFO 4486 --- [ocp-schedules-19,d1d8c54d742c2046,aa1dd0b0129d1a84] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:01:42.012 WARN 4486 --- [scanner-13320229-1,d1d8c54d742c2046,958337daf92266a1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:01:42.084 INFO 4486 --- [alarm-detect-4,d1d8c54d742c2046,d5de1855df855e15] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058475 ~ 1729058492 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058490, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058490, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058490=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:01:42.185 INFO 4486 --- [ocp-schedules-15,604a0218800579b1,2d56b8693050a3a2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:01:42.240 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,fad030046058943e,492019649d47fcc9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fad030046058943e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:42.240 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,fad030046058943e,492019649d47fcc9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fad030046058943e, duration=1 ms] 2024-10-16 14:01:43.028 INFO 4486 --- [ocp-async-5,44d0707db5ba52a7,ebd9da3c2cdd5a9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:01:43.404 INFO 4486 --- [ocp-async-3,c5aabbef333a540b,5f1016f1c9ae1ea2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:43.534 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:43.994 INFO 4486 --- [prometheus-minute-schedule-1,67a8aa4f1afef417,38ff7f6dbc02293e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:44.000 INFO 4486 --- [prometheus-minute-schedule-1,67a8aa4f1afef417,38ff7f6dbc02293e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:44.775 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,015844d8724af58f,41cc49fe30d757a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=015844d8724af58f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:01:44.776 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,015844d8724af58f,41cc49fe30d757a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=015844d8724af58f, duration=2 ms] 2024-10-16 14:01:45.404 INFO 4486 --- [ocp-async-2,610bf0ce8419983b,1b461e107f4974bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:45.594 INFO 4486 --- [ocp-schedules-24,80680e0a078506e8,2ce38af7e4027a8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:01:45.599 INFO 4486 --- [ocp-schedules-24,80680e0a078506e8,2ce38af7e4027a8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:01:45.599 INFO 4486 --- [ocp-schedules-24,80680e0a078506e8,2ce38af7e4027a8c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:01:45.630 INFO 4486 --- [ocp-schedules-12,fd328a57169cad85,06731835fb3ece44] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:01:45.630 INFO 4486 --- [ocp-schedules-12,fd328a57169cad85,06731835fb3ece44] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:01:46.534 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:46.994 INFO 4486 --- [prometheus-minute-schedule-1,1bfd0556fad207c1,b0fd32e9c37c4ed1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:46.999 INFO 4486 --- [prometheus-minute-schedule-1,1bfd0556fad207c1,b0fd32e9c37c4ed1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:47.073 INFO 4486 --- [ocp-schedules-1,28ed5fe509ca9eba,fede4cfe566d8a2d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:01:47.214 INFO 4486 --- [ocp-schedules-26,7701d201749d6bd3,124f4c10c2276ac0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:01:47.267 INFO 4486 --- [ocp-schedules-18,fd886e3898b2d71b,571a5241782f18d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 14:01:47.281 INFO 4486 --- [ocp-schedules-4,5579394691feb9ef,4cbeafdaf026c260] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:01:47.405 INFO 4486 --- [ocp-async-4,1f192f6306e3e6de,9f0d85ef5d8e7577] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:47.980 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,fdef3ee2d8f41e91,91ac9466528feb9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=fdef3ee2d8f41e91, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:01:47.981 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,fdef3ee2d8f41e91,91ac9466528feb9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=fdef3ee2d8f41e91, duration=3 ms] 2024-10-16 14:01:48.387 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,422b4fd9e63d92a4,d751b4605b877dc2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=422b4fd9e63d92a4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:48.387 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,422b4fd9e63d92a4,d751b4605b877dc2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=422b4fd9e63d92a4, duration=1 ms] 2024-10-16 14:01:48.758 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 14:01:49.405 INFO 4486 --- [ocp-async-3,f2735f380d95f115,a6ebbd682c1006db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:49.536 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:49.994 INFO 4486 --- [prometheus-minute-schedule-1,e2d25a86bddd3a5a,cd53cfc24b056949] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:50.000 INFO 4486 --- [prometheus-minute-schedule-1,e2d25a86bddd3a5a,cd53cfc24b056949] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:50.410 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5d92aada39091eb5,920f3e10a85e9a58] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5d92aada39091eb5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:50.411 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5d92aada39091eb5,920f3e10a85e9a58] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5d92aada39091eb5, duration=1 ms] 2024-10-16 14:01:50.565 INFO 4486 --- [ocp-schedules-1,10a663738fcabeb1,e112714875c2dbcf] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:01:50.565 INFO 4486 --- [ocp-schedules-1,10a663738fcabeb1,e112714875c2dbcf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:01:50.567 INFO 4486 --- [ocp-schedules-1,10a663738fcabeb1,e112714875c2dbcf] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:01:50.567 INFO 4486 --- [ocp-schedules-1,10a663738fcabeb1,e112714875c2dbcf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:01:50.568 INFO 4486 --- [ocp-schedules-1,10a663738fcabeb1,e112714875c2dbcf] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:01:50.568 INFO 4486 --- [ocp-schedules-1,10a663738fcabeb1,e112714875c2dbcf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 14:01:51.294 INFO 4486 --- [ocp-schedules-4,e1fb103663721c8a,4a68c548a0ee2343] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:01:51.405 INFO 4486 --- [ocp-async-2,e475c7c916649dbd,73640a43bc17ae31] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:51.513 INFO 4486 --- [ocp-schedules-14,51c6b374692be8a0,9842efd99bd98b35] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:01:51.513 INFO 4486 --- [ocp-schedules-14,51c6b374692be8a0,9842efd99bd98b35] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:01:51.513 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:01:51.513 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:01:51.513 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:01:51.513 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:01:51.513 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:01:51.519 INFO 4486 --- [ocp-schedules-14,51c6b374692be8a0,9842efd99bd98b35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:01:51.519 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:01:51.520 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:01:51.520 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:01:51.520 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:01:51.521 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:01:51.521 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:01:51.521 INFO 4486 --- [ocp-schedules-14,51c6b374692be8a0,9842efd99bd98b35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:01:51.521 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:01:51.522 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:01:51.523 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:01:51.524 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:01:51.545 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:01:51.545 INFO 4486 --- [ocp-schedules-14,51c6b374692be8a0,9842efd99bd98b35] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:01:51.545 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:01:51.547 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:01:51.548 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:01:51.554 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:01:51.557 INFO 4486 --- [ocp-schedules-14,51c6b374692be8a0,9842efd99bd98b35] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:01:51.954 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,1ee25bb48b245786,6524639e4bf3fb39] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1ee25bb48b245786, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:51.954 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,1ee25bb48b245786,6524639e4bf3fb39] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1ee25bb48b245786, duration=1 ms] 2024-10-16 14:01:52.151 INFO 4486 --- [ocp-schedules-26,6f76a169ae0eb9f6,cf4b2086ff5f25d1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:01:52.161 WARN 4486 --- [scanner-13320525-1,6f76a169ae0eb9f6,888186ec953db5f3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:01:52.245 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,fa079f5c85c83fab,df9729fe51266ae2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fa079f5c85c83fab, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:52.245 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,fa079f5c85c83fab,df9729fe51266ae2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fa079f5c85c83fab, duration=1 ms] 2024-10-16 14:01:52.536 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:52.994 INFO 4486 --- [prometheus-minute-schedule-1,094500c4f609e0b2,481b922a2f765a87] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:53.002 INFO 4486 --- [prometheus-minute-schedule-1,094500c4f609e0b2,481b922a2f765a87] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:53.028 INFO 4486 --- [ocp-async-6,3649f63aa78e43fa,ac51c3daf00baa30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:01:53.405 INFO 4486 --- [ocp-async-8,09c98e2a8d54d5d6,f6d8122af98da31c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:55.406 INFO 4486 --- [ocp-async-9,bbaaef0d118d80d3,c3b851ed993c3e59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:55.537 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:55.994 INFO 4486 --- [prometheus-minute-schedule-1,464163956b375e1f,3ea252fa40080ba6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:56.000 INFO 4486 --- [prometheus-minute-schedule-1,464163956b375e1f,3ea252fa40080ba6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:57.406 INFO 4486 --- [ocp-async-6,da06732ae50792b2,467d2a96fc3d0ae0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:01:58.396 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a8973c9bca80ae15,08e89690a78a4d9d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a8973c9bca80ae15, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:01:58.397 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a8973c9bca80ae15,08e89690a78a4d9d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a8973c9bca80ae15, duration=2 ms] 2024-10-16 14:01:58.538 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:01:58.994 INFO 4486 --- [prometheus-minute-schedule-1,6dd19331e18f6a9f,af5b57068c27d161] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058520, interval=60 2024-10-16 14:01:59.001 INFO 4486 --- [prometheus-minute-schedule-1,6dd19331e18f6a9f,af5b57068c27d161] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:01:59.271 INFO 4486 --- [ocp-schedules-17,e42000e84e3bd7d7,8325b7f624b9af3e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:01:59.406 INFO 4486 --- [ocp-async-5,c860298c04e7a5e9,1b0c6be50c7896e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:00.416 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,11acd755acbacb9a,d3b56cde038240c7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=11acd755acbacb9a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:00.416 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,11acd755acbacb9a,d3b56cde038240c7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=11acd755acbacb9a, duration=1 ms] 2024-10-16 14:02:01.299 INFO 4486 --- [ocp-schedules-26,ca59673aa5f52b8d,1a9d94445921725c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:02:01.406 INFO 4486 --- [ocp-async-8,6bd8b36386560140,ffab74a1e8d3412a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:01.539 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:01.964 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,6daee161f5465164,49a8072e8884d860] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6daee161f5465164, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:01.964 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,6daee161f5465164,49a8072e8884d860] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6daee161f5465164, duration=1 ms] 2024-10-16 14:02:01.994 INFO 4486 --- [prometheus-minute-schedule-1,e1bc6b6e2449d629,2c7de1a847cc481a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:02.005 INFO 4486 --- [prometheus-minute-schedule-1,e1bc6b6e2449d629,2c7de1a847cc481a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:02.250 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,53b98b36a48072ca,e97e0c6379b8eb16] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=53b98b36a48072ca, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:02.250 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,53b98b36a48072ca,e97e0c6379b8eb16] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=53b98b36a48072ca, duration=1 ms] 2024-10-16 14:02:02.287 INFO 4486 --- [ocp-schedules-17,5f52c16cebd0c9d0,1f7c8644c321acd2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:02:02.336 WARN 4486 --- [scanner-13320775-1,5f52c16cebd0c9d0,9001163f0b466e7c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:02:02.417 INFO 4486 --- [alarm-detect-16,5f52c16cebd0c9d0,b3a5e61550e127aa] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058490 ~ 1729058512 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058505, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058505, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058505=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:02:02.711 INFO 4486 --- [ocp-schedules-10,4ab6b1f4bfb28702,87497fc7886c6f0d] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:02:02.711+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:02:02.716 INFO 4486 --- [ocp-schedules-10,4ab6b1f4bfb28702,87497fc7886c6f0d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:02:02.729 INFO 4486 --- [ocp-schedules-10,4ab6b1f4bfb28702,87497fc7886c6f0d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:02:02.741 INFO 4486 --- [ocp-schedules-10,4ab6b1f4bfb28702,87497fc7886c6f0d] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:02:03.028 INFO 4486 --- [ocp-async-4,e097e9a9ad1d9f5c,54f1474e66d1df25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:02:03.407 INFO 4486 --- [ocp-async-6,ffc7f4fd8694860a,618f14e92e9ca5e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:04.540 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:04.994 INFO 4486 --- [prometheus-minute-schedule-1,f94adffb8c4c174a,d6dc47ff07066b84] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:05.001 INFO 4486 --- [prometheus-minute-schedule-1,f94adffb8c4c174a,d6dc47ff07066b84] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:05.407 INFO 4486 --- [ocp-async-5,6a53b6dc7d20c1ce,c3d51df5b17c1bca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:07.066 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:02:07.066 INFO 4486 --- [ocp-schedules-5,df498109c890fe7e,528858b89dc65c60] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:02:07.068 INFO 4486 --- [ocp-schedules-5,df498109c890fe7e,528858b89dc65c60] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:07.081 INFO 4486 --- [ocp-schedules-20,c6d278cd25bcf56a,dcb718cf54653d56] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:02:07.083 INFO 4486 --- [ocp-schedules-20,c6d278cd25bcf56a,dcb718cf54653d56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:02:07.089 INFO 4486 --- [ocp-schedules-3,9f37c1d4895bd424,495131983828b9b9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:02:07.089 INFO 4486 --- [ocp-schedules-17,a8c20cc069b85808,0cc793df5b11feed] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:02:07.089 INFO 4486 --- [ocp-schedules-18,2a31af972e986910,d9e18da36e879628] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:02:07.090 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:02:07.090 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:02:07.090 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:02:07.090 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:02:07.091 INFO 4486 --- [host-check-agent-status1,2a31af972e986910,e6562ee569e9c5c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:02:07.091 INFO 4486 --- [host-check-clockdiff1,a8c20cc069b85808,26c2b47159cc92e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:02:07.091 INFO 4486 --- [host-check-clockdiff2,a8c20cc069b85808,8bf88eba0a65a000] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:02:07.091 INFO 4486 --- [host-check-clockdiff3,a8c20cc069b85808,40f215d8e16ad175] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:02:07.092 INFO 4486 --- [host-check-clockdiff4,a8c20cc069b85808,2a66b87c58bad582] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:02:07.092 INFO 4486 --- [host-check-agent-status2,2a31af972e986910,e68a5e5e6a72a4cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:02:07.092 INFO 4486 --- [host-check-agent-status3,2a31af972e986910,e4a3d96eff6f749e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:02:07.093 INFO 4486 --- [host-check-agent-status4,2a31af972e986910,0de5834f448ba411] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:02:07.097 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:07.098 INFO 4486 --- [host-check-agent-status1,2a31af972e986910,e6562ee569e9c5c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:07.098 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:02:07.099 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:02:07.099 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:02:07.100 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:02:07.100 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:07.100 INFO 4486 --- [host-check-agent-status1,2a31af972e986910,e6562ee569e9c5c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:07.100 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:02:07.110 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:07.120 INFO 4486 --- [host-check-agent-status4,2a31af972e986910,0de5834f448ba411] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:02:07.105 INFO 4486 --- [host-check-agent-status3,2a31af972e986910,e4a3d96eff6f749e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:02:07.123 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:02:07.123 INFO 4486 --- [host-check-clockdiff1,a8c20cc069b85808,26c2b47159cc92e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058527 0 0 , error= 2024-10-16 14:02:07.132 INFO 4486 --- [host-check-agent-status2,2a31af972e986910,e68a5e5e6a72a4cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:02:07.134 INFO 4486 --- [host-check-clockdiff2,a8c20cc069b85808,8bf88eba0a65a000] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058527 4 4 , error= 2024-10-16 14:02:07.135 INFO 4486 --- [host-check-agent-status3,2a31af972e986910,e4a3d96eff6f749e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:02:07.135 INFO 4486 --- [host-check-agent-status2,2a31af972e986910,e68a5e5e6a72a4cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:02:07.135 INFO 4486 --- [host-check-agent-status4,2a31af972e986910,0de5834f448ba411] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:02:07.138 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.138 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:07.139 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:02:07.156 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:07.158 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.160 INFO 4486 --- [host-check-agent-status1,2a31af972e986910,e6562ee569e9c5c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:02:07.162 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:02:07.162 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:02:07.162 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.164 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:07.164 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:02:07.164 INFO 4486 --- [host-check-agent-status4,2a31af972e986910,0de5834f448ba411] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:02:07.164 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:02:07.166 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:02:07.164558528+08:00 2024-10-16 14:02:07.166 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:02:07.166 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:02:07.166+08:00 2024-10-16 14:02:07.166 INFO 4486 --- [host-check-agent-status1,2a31af972e986910,e6562ee569e9c5c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:02:07.167 INFO 4486 --- [host-check-agent-status1,2a31af972e986910,e6562ee569e9c5c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:02:07.167 INFO 4486 --- [host-check-agent-status1,2a31af972e986910,e6562ee569e9c5c5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:02:07.167 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:02:07.170714402+08:00 2024-10-16 14:02:07.167 INFO 4486 --- [host-check-agent-status3,2a31af972e986910,e4a3d96eff6f749e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:02:07.167 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:02:07.170689414+08:00 2024-10-16 14:02:07.167 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:02:07.167+08:00 2024-10-16 14:02:07.168 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:02:07.167+08:00 2024-10-16 14:02:07.169 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:02:07.172814111+08:00 2024-10-16 14:02:07.169 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:02:07.166+08:00 and status AVAILABLE 2024-10-16 14:02:07.169 INFO 4486 --- [host-check-agent-status2,2a31af972e986910,e68a5e5e6a72a4cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:02:07.169 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.169 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:02:07.169 INFO 4486 --- [host-check-agent-status4,2a31af972e986910,0de5834f448ba411] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:02:07.170 INFO 4486 --- [host-check-agent-status4,2a31af972e986910,0de5834f448ba411] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:02:07.170 INFO 4486 --- [host-check-agent-status4,2a31af972e986910,0de5834f448ba411] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:02:07.170 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:02:07.169+08:00 2024-10-16 14:02:07.170 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:02:07.167+08:00 and status AVAILABLE 2024-10-16 14:02:07.170 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:02:07.167+08:00 and status AVAILABLE 2024-10-16 14:02:07.170 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:07.172 INFO 4486 --- [host-check-agent-status3,2a31af972e986910,e4a3d96eff6f749e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:02:07.172 INFO 4486 --- [host-check-agent-status3,2a31af972e986910,e4a3d96eff6f749e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:02:07.172 INFO 4486 --- [host-check-agent-status3,2a31af972e986910,e4a3d96eff6f749e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:02:07.172 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:02:07.172 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:02:07.172 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:02:07.169+08:00 and status AVAILABLE 2024-10-16 14:02:07.172 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:02:07.174 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:02:07.175 INFO 4486 --- [host-check-agent-status2,2a31af972e986910,e68a5e5e6a72a4cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:02:07.175 INFO 4486 --- [host-check-agent-status2,2a31af972e986910,e68a5e5e6a72a4cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:02:07.175 INFO 4486 --- [host-check-agent-status2,2a31af972e986910,e68a5e5e6a72a4cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:02:07.180 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.180 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:02:07.181 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:07.181 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:02:07.181 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:07.182 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:02:07.182 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:02:07.182 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:07.182 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:02:07.183 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:02:07.190 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.191 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:07.193 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.194 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:02:07.203 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:07.204 INFO 4486 --- [ocp-schedules-6,b584fea461299abf,04dbe117f4545cf7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:02:07.204 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:07.205 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:07.205 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:07.206 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:07.234 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:02:07.239 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:02:07.243 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:02:07.251 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:02:07.251 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:02:07.255 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:02:07.261 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:02:07.269 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:07.270 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:07.272 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:07.273 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:02:07.288 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:07.297 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:02:07.306 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:02:07.308 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:02:07.313 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:02:07.317 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:02:07.326 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:07.326 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:02:07.334 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:07.344 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:07.351 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:02:07.353 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:02:07.363 INFO 4486 --- [host-check-host-status3,9f37c1d4895bd424,334fd25ba5702bc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:02:07.363 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:02:07.365 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:02:07.377 INFO 4486 --- [host-check-host-status4,9f37c1d4895bd424,8155072d18f01211] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:02:07.379 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:02:07.381 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:07.391 INFO 4486 --- [host-check-host-status2,9f37c1d4895bd424,b7ce9651d718ec80] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:02:07.407 INFO 4486 --- [ocp-async-4,c047a8e064ebc629,8e672535330b26f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:07.435 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:02:07.444 INFO 4486 --- [host-check-host-status1,9f37c1d4895bd424,93cf6f505a61312b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:02:07.541 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:07.994 INFO 4486 --- [prometheus-minute-schedule-1,4ca6169cd853d28e,9e1f3e2c9372da9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:08.002 INFO 4486 --- [prometheus-minute-schedule-1,4ca6169cd853d28e,9e1f3e2c9372da9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:08.052 INFO 4486 --- [host-check-clockdiff3,a8c20cc069b85808,40f215d8e16ad175] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058528 4 4 , error= 2024-10-16 14:02:08.061 INFO 4486 --- [host-check-clockdiff4,a8c20cc069b85808,2a66b87c58bad582] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058528 4 4 , error= 2024-10-16 14:02:08.410 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6d22029ae0eb0674,ec88cfd73214aebf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6d22029ae0eb0674, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:08.410 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6d22029ae0eb0674,ec88cfd73214aebf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6d22029ae0eb0674, duration=2 ms] 2024-10-16 14:02:09.407 INFO 4486 --- [ocp-async-3,f3df29693b218ff0,f098f959205015f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:10.007 INFO 4486 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:02:10.007 INFO 4486 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:02:10.025 INFO 4486 --- [Check obproxy1,a806a2a99fb2411a,b0af881485b05084] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:02:10.025 INFO 4486 --- [Check obproxy1,a806a2a99fb2411a,b0af881485b05084] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:10.027 INFO 4486 --- [Check obproxy2,575193b48befe75c,1ce6214f0acf393c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:02:10.027 INFO 4486 --- [Check obproxy2,575193b48befe75c,1ce6214f0acf393c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:10.032 INFO 4486 --- [Collect obproxy2,6a6327a941d74d91,8fb5fb0cfc4fb641] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:02:10.032 INFO 4486 --- [Collect obproxy1,40af6a2d9398d2f9,cc20c333067b6b10] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:02:10.032 INFO 4486 --- [Collect obproxy1,40af6a2d9398d2f9,cc20c333067b6b10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:10.032 INFO 4486 --- [Collect obproxy2,6a6327a941d74d91,8fb5fb0cfc4fb641] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:10.051 INFO 4486 --- [Check obproxy2,575193b48befe75c,1ce6214f0acf393c] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:02:10.051 INFO 4486 --- [Check obproxy2,575193b48befe75c,1ce6214f0acf393c] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:02:10.052 INFO 4486 --- [Collect obproxy2,6a6327a941d74d91,8fb5fb0cfc4fb641] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:02:10.054 INFO 4486 --- [Check obproxy1,a806a2a99fb2411a,b0af881485b05084] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:02:10.054 INFO 4486 --- [Check obproxy1,a806a2a99fb2411a,b0af881485b05084] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:02:10.057 INFO 4486 --- [Collect obproxy2,6a6327a941d74d91,8fb5fb0cfc4fb641] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:02:10.060 INFO 4486 --- [Collect obproxy1,40af6a2d9398d2f9,cc20c333067b6b10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:02:10.061 INFO 4486 --- [Collect obproxy2,6a6327a941d74d91,8fb5fb0cfc4fb641] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:02:10.066 INFO 4486 --- [Collect obproxy1,40af6a2d9398d2f9,cc20c333067b6b10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:02:10.072 INFO 4486 --- [Collect obproxy1,40af6a2d9398d2f9,cc20c333067b6b10] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:02:10.420 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ef233736efa486c7,4cce9a2c7781c601] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ef233736efa486c7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:10.420 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,ef233736efa486c7,4cce9a2c7781c601] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ef233736efa486c7, duration=1 ms] 2024-10-16 14:02:10.510 INFO 4486 --- [ocp-schedules-16,ef6b8cbd57588c69,d55ba5601b23c7d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:02:10.510 INFO 4486 --- [ocp-schedules-16,23fb5aca4b361d50,842ab988cfe1a65b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:02:10.512 INFO 4486 --- [ocp-schedules-16,23fb5aca4b361d50,842ab988cfe1a65b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:10.541 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.541 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:10.541 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:10.553 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.554 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:10.571 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.572 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:10.576 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:10.577 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.577 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:02:10.610 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:10.611 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.611 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:10.612 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:02:10.613 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.616 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:02:10.617 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:10.626 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.626 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:10.638 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.647 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:10.649 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.649 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:10.650 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:02:10.655 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:02:10.660 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:10.662 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:10.685 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:02:10.688 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:10.689 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.690 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:02:10.733 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:02:10.733 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:02:10.744 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:10.746 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.747 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:10.751 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.754 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:02:10.759 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:10.764 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:02:10.773 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:10.774 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.775 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:02:10.786 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:02:10.786 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:02:10.787 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.787 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:10.813 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.813 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:10.836 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.836 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:10.864 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.864 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:10.890 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.890 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:10.915 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:10.941 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:10.942 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:10.943 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:10.947 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:02:10.952 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:02:10.957 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:02:10.959 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:02:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:02:10.980 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:02:10.994 INFO 4486 --- [prometheus-minute-schedule-1,52d1584db84859cf,a22fa64a3cc45f8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:11.001 INFO 4486 --- [prometheus-minute-schedule-1,52d1584db84859cf,a22fa64a3cc45f8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:02:11.143 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:02:11.202 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:11.203 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:11.204 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:02:11.260 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:02:11.260 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:02:11.309 INFO 4486 --- [ocp-schedules-29,c310cb51ad3ef2c9,e9dc4f940af98f85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:02:11.408 INFO 4486 --- [ocp-async-2,24475c18ce3257e6,571ff5aad8e230e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:11.980 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f9758d226bb49aa4,b56ca1892ae61a63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f9758d226bb49aa4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:11.980 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f9758d226bb49aa4,b56ca1892ae61a63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f9758d226bb49aa4, duration=1 ms] 2024-10-16 14:02:12.278 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c6dcf327d2a81c69,ddfd8d541ae7f4de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c6dcf327d2a81c69, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:12.278 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,c6dcf327d2a81c69,ddfd8d541ae7f4de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c6dcf327d2a81c69, duration=1 ms] 2024-10-16 14:02:12.557 INFO 4486 --- [ocp-schedules-12,d2a81a38c401b6f4,03f66fb60c8cafc5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:02:12.594 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,36d481d325737d34,f88512e002ce3883] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=36d481d325737d34, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:02:12.601 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,36d481d325737d34,f88512e002ce3883] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=36d481d325737d34, duration=10 ms] 2024-10-16 14:02:12.603 WARN 4486 --- [scanner-13321981-1,d2a81a38c401b6f4,109b31fe4356a4f5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:02:12.658 INFO 4486 --- [alarm-detect-2,d2a81a38c401b6f4,1aac11c841dda140] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058505 ~ 1729058522 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058520, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058520, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058520=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:02:13.029 INFO 4486 --- [ocp-async-1,a8181d5e84baa29b,c7d4a66cdb086239] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:02:13.408 INFO 4486 --- [ocp-async-4,47f1ed73bdf6a22c,f739fbcf2dac9d35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:13.542 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:13.994 INFO 4486 --- [prometheus-minute-schedule-1,e42e88f44dd5af16,622cb18f34b034a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:13.999 INFO 4486 --- [prometheus-minute-schedule-1,e42e88f44dd5af16,622cb18f34b034a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:14.552 ERROR 4486 --- [metric-archive-second-20,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1694585) Duplicate entry '586774769061924864-1729058460' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:02:14.551210] [YB420AC531B8-00060B7DA1ABF118-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1694585) Duplicate entry '586774769061924864-1729058460' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:02:14.551210] [YB420AC531B8-00060B7DA1ABF118-0-0] 2024-10-16 14:02:14.579 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058460, nodeCount=1000, affectRows=1000 2024-10-16 14:02:14.580 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058460, nodeCount=1000, affectRows=1000 2024-10-16 14:02:14.582 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058460, nodeCount=1000, affectRows=1000 2024-10-16 14:02:14.583 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058460, nodeCount=1000, validCount=1000 2024-10-16 14:02:14.585 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058460, nodeCount=1000, validCount=1000 2024-10-16 14:02:14.586 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058460, nodeCount=999, validCount=999 2024-10-16 14:02:14.759 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058460, nodeCount=449, affectRows=449 2024-10-16 14:02:14.761 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058460, nodeCount=447, validCount=447 2024-10-16 14:02:15.219 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058460, nodeCount=477, affectRows=477 2024-10-16 14:02:15.408 INFO 4486 --- [ocp-async-3,cc1e7ddd44608a4b,433ed27d4a2ba566] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:15.886 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058460, nodeCount=477, validCount=477 2024-10-16 14:02:16.543 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:16.648 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 14:02:16.649 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 14:02:16.657 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 14:02:16.657 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 14:02:16.657 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 14:02:16.657 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 14:02:16.663 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 14:02:16.663 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 14:02:16.663 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 14:02:16.663 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 14:02:16.709 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 14:02:16.716 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 14:02:16.717 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 14:02:16.717 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 14:02:16.717 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 14:02:16.717 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 14:02:16.724 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 14:02:16.994 INFO 4486 --- [prometheus-minute-schedule-1,044d9dc0ca7aed5b,6483c81183685bb4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:17.001 INFO 4486 --- [prometheus-minute-schedule-1,044d9dc0ca7aed5b,6483c81183685bb4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:17.069 INFO 4486 --- [ocp-schedules-5,df498109c890fe7e,528858b89dc65c60] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:02:17.170 INFO 4486 --- [ocp-schedules-20,c6d278cd25bcf56a,dcb718cf54653d56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:02:17.209 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 14:02:17.284 INFO 4486 --- [ocp-schedules-1,64ba0887f6077686,d9c9eafe3bb4efdd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:02:17.410 INFO 4486 --- [ocp-async-1,eccaa55eeb485c9e,3adefed820e9384e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:17.703 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.040584188S 2024-10-16 14:02:17.704 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 14:02:17.773 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 14:02:17.773 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 14:02:17.795 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 14:02:17.868 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 14:02:17.868 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 14:02:17.869 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 14:02:17.869 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 14:02:17.869 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 14:02:17.869 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 14:02:17.876 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 14:02:17.997 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] o.s.boot.SpringApplication : Started application in 1.339 seconds (JVM running for 79441.048) 2024-10-16 14:02:17.997 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.340007779S 2024-10-16 14:02:18.332 INFO 4486 --- [ocp-schedules-28,cea9e2fabc93dd5f,3b3e84c082becdfc] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 14:02:18.416 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,5864008041fa3ede,c4cfe156d4696dde] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5864008041fa3ede, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:18.416 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,5864008041fa3ede,c4cfe156d4696dde] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5864008041fa3ede, duration=3 ms] 2024-10-16 14:02:19.277 INFO 4486 --- [ocp-schedules-29,f4b116426554f5b2,063c4cdb1d04652b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:02:19.410 INFO 4486 --- [ocp-async-5,beb8437ce2df3c6c,07fabebdaca62c79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:19.543 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:19.994 INFO 4486 --- [prometheus-minute-schedule-1,796c8d09119c6481,6f39bb08bd02b113] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:19.999 INFO 4486 --- [prometheus-minute-schedule-1,796c8d09119c6481,6f39bb08bd02b113] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:20.425 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,28df60505dfac1a5,bcc64c31e401b10a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=28df60505dfac1a5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:20.425 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,28df60505dfac1a5,bcc64c31e401b10a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=28df60505dfac1a5, duration=1 ms] 2024-10-16 14:02:20.814 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 14:02:20.870 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 14:02:20.870 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 14:02:20.996 INFO 4486 --- [prometheus-cache-monitor1,969e573042524696,8503294121bcd001] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":16597} 2024-10-16 14:02:20.996 INFO 4486 --- [prometheus-cache-monitor1,969e573042524696,8503294121bcd001] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:02:21.319 INFO 4486 --- [ocp-schedules-3,8336c15d377af8cd,8abe9e077c98badc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:02:21.411 INFO 4486 --- [ocp-async-4,f676a86045060580,87a98f84734e0fdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:21.984 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5c355e32e7b32522,ed1bf35bf2fac016] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5c355e32e7b32522, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:21.985 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,5c355e32e7b32522,ed1bf35bf2fac016] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5c355e32e7b32522, duration=2 ms] 2024-10-16 14:02:22.282 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,59e1cd05d2242d64,5aeab0d4c183643b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=59e1cd05d2242d64, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:22.282 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,59e1cd05d2242d64,5aeab0d4c183643b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=59e1cd05d2242d64, duration=1 ms] 2024-10-16 14:02:22.545 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:22.736 INFO 4486 --- [ocp-schedules-26,7559e0a112f52c29,7f4c9b6a06838599] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:02:22.791 WARN 4486 --- [scanner-13322202-1,7559e0a112f52c29,74bd5228d814606b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:02:22.994 INFO 4486 --- [prometheus-minute-schedule-1,1694d60d94f82b0d,e2b2854145a1afac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:23.000 INFO 4486 --- [prometheus-minute-schedule-1,1694d60d94f82b0d,e2b2854145a1afac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:23.029 INFO 4486 --- [ocp-async-5,2a65206b627c5c1e,e0c3826efd7bafca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:02:23.411 INFO 4486 --- [ocp-async-2,a9a80d850a0bd833,73ad55065c495fda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:23.581 INFO 4486 --- [ocp-schedules-20,be09452308f4b759,a2ae10867fb84314] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:02:23.582 INFO 4486 --- [ocp-schedules-20,be09452308f4b759,a2ae10867fb84314] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:23.583 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:02:23.583 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:02:23.583 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:02:23.961 INFO 4486 --- [ocp-schedules-29,f68a31d3c3a58959,484766050ec97a31] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:02:25.246 INFO 4486 --- [ocp-schedules-28,710f5c4c1628380e,d4c7e45b61fec11c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:02:25.411 INFO 4486 --- [ocp-async-10,6b210d8b390df088,1f9f97b13d3cc0bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:25.545 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:25.994 INFO 4486 --- [prometheus-minute-schedule-1,3f910c645b9c9534,415a987341174420] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:26.000 INFO 4486 --- [prometheus-minute-schedule-1,3f910c645b9c9534,415a987341174420] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:26.164 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,19e1122dd298459c,5b764ad16cf41b90] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=19e1122dd298459c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:02:26.164 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,19e1122dd298459c,5b764ad16cf41b90] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=19e1122dd298459c, duration=2 ms] 2024-10-16 14:02:26.852 INFO 4486 --- [ocp-schedules-5,7733dd889cb94fd1,1709189fb3931219] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:02:26.853 INFO 4486 --- [ocp-schedules-5,7733dd889cb94fd1,1709189fb3931219] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:02:26.888 INFO 4486 --- [Collect all tenants major compactions1,7733dd889cb94fd1,9cd09c1601d96ffc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:02:26.889 INFO 4486 --- [Collect all tenants major compactions2,7733dd889cb94fd1,668f8b4721ae7fcf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:02:26.889 INFO 4486 --- [Collect all tenants major compactions3,7733dd889cb94fd1,031fbe42b37a4b48] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:02:26.890 INFO 4486 --- [Collect all tenants major compactions4,7733dd889cb94fd1,3f3792b8d3153d1e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:02:26.890 INFO 4486 --- [Collect all tenants major compactions5,7733dd889cb94fd1,e6b3660b2bdef0e8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:02:26.906 INFO 4486 --- [Collect all tenants major compactions1,7733dd889cb94fd1,9cd09c1601d96ffc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:26.906 INFO 4486 --- [Collect all tenants major compactions3,7733dd889cb94fd1,031fbe42b37a4b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:26.907 INFO 4486 --- [Collect all tenants major compactions2,7733dd889cb94fd1,668f8b4721ae7fcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:26.907 INFO 4486 --- [Collect all tenants major compactions1,7733dd889cb94fd1,9cd09c1601d96ffc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:02:26.908 INFO 4486 --- [Collect all tenants major compactions2,7733dd889cb94fd1,668f8b4721ae7fcf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:02:26.909 INFO 4486 --- [Collect all tenants major compactions3,7733dd889cb94fd1,031fbe42b37a4b48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:02:26.911 INFO 4486 --- [Collect all tenants major compactions2,7733dd889cb94fd1,668f8b4721ae7fcf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:02:26.912 INFO 4486 --- [Collect all tenants major compactions1,7733dd889cb94fd1,9cd09c1601d96ffc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:02:26.913 INFO 4486 --- [Collect all tenants major compactions3,7733dd889cb94fd1,031fbe42b37a4b48] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:02:26.916 INFO 4486 --- [Collect all tenants major compactions2,7733dd889cb94fd1,668f8b4721ae7fcf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:02:26.919 INFO 4486 --- [Collect all tenants major compactions4,7733dd889cb94fd1,3f3792b8d3153d1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:26.919 INFO 4486 --- [Collect all tenants major compactions5,7733dd889cb94fd1,e6b3660b2bdef0e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:26.920 INFO 4486 --- [Collect all tenants major compactions4,7733dd889cb94fd1,3f3792b8d3153d1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:02:26.921 INFO 4486 --- [Collect all tenants major compactions5,7733dd889cb94fd1,e6b3660b2bdef0e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:02:26.922 INFO 4486 --- [Collect all tenants major compactions3,7733dd889cb94fd1,031fbe42b37a4b48] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:02:26.923 INFO 4486 --- [Collect all tenants major compactions1,7733dd889cb94fd1,9cd09c1601d96ffc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:02:26.924 INFO 4486 --- [Collect all tenants major compactions4,7733dd889cb94fd1,3f3792b8d3153d1e] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:02:26.924 INFO 4486 --- [Collect all tenants major compactions4,7733dd889cb94fd1,3f3792b8d3153d1e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:02:26.924 INFO 4486 --- [Collect all tenants major compactions5,7733dd889cb94fd1,e6b3660b2bdef0e8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:02:26.924 INFO 4486 --- [Collect all tenants major compactions5,7733dd889cb94fd1,e6b3660b2bdef0e8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:02:26.926 INFO 4486 --- [ocp-schedules-5,7733dd889cb94fd1,1709189fb3931219] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:02:27.411 INFO 4486 --- [ocp-async-5,f211004833855fa1,638f31abfbde0fa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:27.903 INFO 4486 --- [ocp-schedules-25,e4cc7323fcd5c2e0,ad8d962566365342] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:02:27.903+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:02:27.905 INFO 4486 --- [ocp-schedules-25,e4cc7323fcd5c2e0,ad8d962566365342] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:02:27.905 INFO 4486 --- [ocp-schedules-25,e4cc7323fcd5c2e0,ad8d962566365342] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:02:27.906 INFO 4486 --- [ocp-schedules-25,e4cc7323fcd5c2e0,ad8d962566365342] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:02:28.420 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f1c9f0d512cee16c,3de50992731786fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f1c9f0d512cee16c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:28.421 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f1c9f0d512cee16c,3de50992731786fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f1c9f0d512cee16c, duration=1 ms] 2024-10-16 14:02:28.546 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:28.994 INFO 4486 --- [prometheus-minute-schedule-1,9494a695c4f7593d,59d0d911716250c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:29.001 INFO 4486 --- [prometheus-minute-schedule-1,9494a695c4f7593d,59d0d911716250c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:29.162 INFO 4486 --- [ocp-schedules-2,5a1999ba80c093d5,7be90016820e570d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 14:02:29.164 INFO 4486 --- [ocp-schedules-2,5a1999ba80c093d5,7be90016820e570d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:29.207 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:02:29.208 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.213 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:02:29.229 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:02:29.229 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.229 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.230 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:02:29.233 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:02:29.241 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.242 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:02:29.253 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.253 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:02:29.254 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.255 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:02:29.267 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.268 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:02:29.269 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.270 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:02:29.270 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.270 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:02:29.275 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:02:29.277 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.277 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:29.285 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.286 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:29.287 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.287 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.288 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:29.288 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:02:29.298 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.298 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.298 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:02:29.299 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:29.302 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.303 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:02:29.308 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.308 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:02:29.309 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.309 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:02:29.317 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.318 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:02:29.319 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.320 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:02:29.322 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.322 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:02:29.330 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.331 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.332 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.332 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.333 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:02:29.333 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.334 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.335 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.337 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.338 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.338 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.339 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.340 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.340 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.341 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.343 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.343 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.343 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.344 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:02:29.344 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.345 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.346 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.346 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.346 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.347 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:29.347 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.347 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.348 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.349 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.349 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.350 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.351 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.352 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 14:02:29.352 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.354 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.356 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.357 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.358 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:29.358 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.359 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:02:29.361 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.362 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.367 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.367 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.368 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:29.373 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.378 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.378 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:29.379 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.380 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.380 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:02:29.383 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.384 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.388 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.390 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.392 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.393 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 14:02:29.395 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 14:02:29.398 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:02:29.398 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 14:02:29.398 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:02:29.405 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.406 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.410 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.411 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.412 INFO 4486 --- [ocp-async-6,c3845039a13fdd8e,4d3e3611d0566886] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:29.414 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.415 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.418 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.419 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.422 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.423 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.426 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.427 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.430 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:29.430 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.431 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.432 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:02:29.434 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:02:29.435 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.436 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:02:29.439 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.441 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:02:29.444 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 14:02:29.447 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.447 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 14:02:29.447 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.447 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 14:02:29.447 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.447 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 14:02:29.447 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:02:29.450 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 14:02:29.450 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 14:02:29.523 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:29.524 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:29.525 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:02:29.537 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.537 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 14:02:29.537 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.537 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 14:02:29.537 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:02:29.537 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 14:02:29.543 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 14:02:29.543 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:02:30.103 INFO 4486 --- [prometheus-second-schedule-1,0d1be30cbf0bdffe,1c59a8a6d3d7640f] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:02:30.430 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,447264432bcc643a,00f872a17d1b47b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=447264432bcc643a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:30.430 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,447264432bcc643a,00f872a17d1b47b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=447264432bcc643a, duration=2 ms] 2024-10-16 14:02:31.324 INFO 4486 --- [ocp-schedules-5,456df6414805007d,2bd640b70d261ecd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:02:31.412 INFO 4486 --- [ocp-async-1,a2164aa853916d4f,ac78c1b1eea5b2ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:31.547 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:31.991 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,affc3b3b8d7d5c35,e59970a405965712] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=affc3b3b8d7d5c35, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:31.991 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,affc3b3b8d7d5c35,e59970a405965712] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=affc3b3b8d7d5c35, duration=2 ms] 2024-10-16 14:02:31.994 INFO 4486 --- [prometheus-minute-schedule-1,aa147d4643096e4c,ddeb230bcabccf9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:32.000 INFO 4486 --- [prometheus-minute-schedule-1,aa147d4643096e4c,ddeb230bcabccf9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:32.286 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5bd6763565f8792d,30e7957da3c9ed68] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5bd6763565f8792d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:32.286 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5bd6763565f8792d,30e7957da3c9ed68] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5bd6763565f8792d, duration=1 ms] 2024-10-16 14:02:32.926 INFO 4486 --- [ocp-schedules-19,7ae2981a947d3725,1077ead9db63844d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:02:32.947 WARN 4486 --- [scanner-13322500-1,7ae2981a947d3725,be9c40ae22dc4f78] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:02:33.031 INFO 4486 --- [ocp-async-2,f28c3eb7d23c817c,4b1848710d36fa6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:02:33.031 INFO 4486 --- [alarm-detect-2,7ae2981a947d3725,5fea879674460e3c] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058520 ~ 1729058542 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058535, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058535, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058535=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:02:33.412 INFO 4486 --- [ocp-async-9,483702c4a3f996e5,2f3fd51c21bd0df7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:33.583 INFO 4486 --- [ocp-schedules-20,be09452308f4b759,a2ae10867fb84314] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:02:34.548 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:34.994 INFO 4486 --- [prometheus-minute-schedule-1,985c73d42b9ebf70,1a06a0e73a403db1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:35.000 INFO 4486 --- [prometheus-minute-schedule-1,985c73d42b9ebf70,1a06a0e73a403db1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:35.412 INFO 4486 --- [ocp-async-3,b866e5432d8e3a30,63ad7c3107a80db4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:37.063 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:02:37.064 INFO 4486 --- [ocp-schedules-20,a2a076e51645f301,ffdc68c27a622783] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:02:37.066 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:02:37.067 INFO 4486 --- [ocp-schedules-19,faf0b87decbd50b5,e295c85ae9b7d753] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:02:37.068 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:02:37.068 INFO 4486 --- [ocp-schedules-19,faf0b87decbd50b5,e295c85ae9b7d753] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:37.070 INFO 4486 --- [ocp-schedules-20,a2a076e51645f301,ffdc68c27a622783] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:02:37.081 INFO 4486 --- [ocp-schedules-14,555c8a863737c37a,a2d8ce0822a99e9d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:02:37.083 INFO 4486 --- [ocp-schedules-14,555c8a863737c37a,a2d8ce0822a99e9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:02:37.089 INFO 4486 --- [ocp-schedules-23,ac058eb9b23e97a5,defeca7e781c2c0a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:02:37.089 INFO 4486 --- [ocp-schedules-8,4abf702de83233a9,fa93dd09494f4a54] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:02:37.090 INFO 4486 --- [ocp-schedules-5,0a4dfe41de421a80,55b38704805b1138] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:02:37.090 INFO 4486 --- [host-check-clockdiff1,ac058eb9b23e97a5,f2d2e9c966b4a788] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:02:37.090 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:02:37.090 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:02:37.091 INFO 4486 --- [host-check-clockdiff2,ac058eb9b23e97a5,61e99e045b934754] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:02:37.091 INFO 4486 --- [host-check-agent-status1,0a4dfe41de421a80,756c3d66710c6157] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:02:37.092 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:02:37.092 INFO 4486 --- [host-check-clockdiff3,ac058eb9b23e97a5,e05930e5eee24759] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:02:37.093 INFO 4486 --- [host-check-agent-status3,0a4dfe41de421a80,86ab8cd3da93db6b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:02:37.093 INFO 4486 --- [host-check-agent-status4,0a4dfe41de421a80,7fcfef88d17f05a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:02:37.093 INFO 4486 --- [host-check-clockdiff4,ac058eb9b23e97a5,d091569d813e8b7b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:02:37.093 INFO 4486 --- [host-check-agent-status2,0a4dfe41de421a80,1dba56cf73f6e998] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:02:37.095 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:02:37.100 INFO 4486 --- [host-check-agent-status3,0a4dfe41de421a80,86ab8cd3da93db6b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:02:37.101 INFO 4486 --- [host-check-agent-status2,0a4dfe41de421a80,1dba56cf73f6e998] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:02:37.101 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:37.101 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:02:37.101 INFO 4486 --- [host-check-agent-status1,0a4dfe41de421a80,756c3d66710c6157] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:37.102 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:02:37.102 INFO 4486 --- [host-check-agent-status4,0a4dfe41de421a80,7fcfef88d17f05a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:02:37.102 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:02:37.113 INFO 4486 --- [host-check-agent-status3,0a4dfe41de421a80,86ab8cd3da93db6b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:02:37.113 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:02:37.113 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:37.114 INFO 4486 --- [host-check-agent-status2,0a4dfe41de421a80,1dba56cf73f6e998] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:02:37.114 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:02:37.114 INFO 4486 --- [host-check-agent-status4,0a4dfe41de421a80,7fcfef88d17f05a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:02:37.115 INFO 4486 --- [host-check-clockdiff3,ac058eb9b23e97a5,e05930e5eee24759] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058557 4 4 , error= 2024-10-16 14:02:37.115 INFO 4486 --- [host-check-clockdiff4,ac058eb9b23e97a5,d091569d813e8b7b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058557 4 4 , error= 2024-10-16 14:02:37.115 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:02:37.129 INFO 4486 --- [host-check-agent-status1,0a4dfe41de421a80,756c3d66710c6157] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:37.129 INFO 4486 --- [host-check-clockdiff1,ac058eb9b23e97a5,f2d2e9c966b4a788] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058557 0 0 , error= 2024-10-16 14:02:37.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:37.141 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:02:37.155 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.155 INFO 4486 --- [host-check-agent-status2,0a4dfe41de421a80,1dba56cf73f6e998] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:02:37.156 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:37.158 INFO 4486 --- [host-check-agent-status3,0a4dfe41de421a80,86ab8cd3da93db6b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:02:37.158 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:02:37.158 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:02:37.158 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:02:37.159 INFO 4486 --- [host-check-agent-status1,0a4dfe41de421a80,756c3d66710c6157] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:02:37.159 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:02:37.160 INFO 4486 --- [host-check-agent-status4,0a4dfe41de421a80,7fcfef88d17f05a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:02:37.160 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:02:37.160 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:02:37.164307083+08:00 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:02:37.164702585+08:00 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:02:37.160853600+08:00 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:02:37.160+08:00 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:02:37.165403097+08:00 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:02:37.161+08:00 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-agent-status2,0a4dfe41de421a80,1dba56cf73f6e998] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-agent-status2,0a4dfe41de421a80,1dba56cf73f6e998] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:02:37.161 INFO 4486 --- [host-check-agent-status2,0a4dfe41de421a80,1dba56cf73f6e998] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:02:37.162 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:02:37.161+08:00 2024-10-16 14:02:37.162 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:02:37.161+08:00 2024-10-16 14:02:37.163 INFO 4486 --- [host-check-agent-status3,0a4dfe41de421a80,86ab8cd3da93db6b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:02:37.163 INFO 4486 --- [host-check-agent-status3,0a4dfe41de421a80,86ab8cd3da93db6b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:02:37.163 INFO 4486 --- [host-check-agent-status3,0a4dfe41de421a80,86ab8cd3da93db6b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:02:37.165 INFO 4486 --- [host-check-agent-status1,0a4dfe41de421a80,756c3d66710c6157] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:02:37.165 INFO 4486 --- [host-check-agent-status1,0a4dfe41de421a80,756c3d66710c6157] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:02:37.165 INFO 4486 --- [host-check-agent-status1,0a4dfe41de421a80,756c3d66710c6157] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:02:37.172 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.172 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:02:37.161+08:00 and status AVAILABLE 2024-10-16 14:02:37.173 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:02:37.161+08:00 and status AVAILABLE 2024-10-16 14:02:37.173 INFO 4486 --- [host-check-agent-status4,0a4dfe41de421a80,7fcfef88d17f05a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:02:37.173 INFO 4486 --- [host-check-agent-status4,0a4dfe41de421a80,7fcfef88d17f05a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:02:37.173 INFO 4486 --- [host-check-agent-status4,0a4dfe41de421a80,7fcfef88d17f05a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:02:37.174 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:02:37.175 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:02:37.175 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.176 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:02:37.161+08:00 and status AVAILABLE 2024-10-16 14:02:37.176 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:02:37.177 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:02:37.177 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:02:37.160+08:00 and status AVAILABLE 2024-10-16 14:02:37.178 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:02:37.178 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.179 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.179 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:02:37.180 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:02:37.183 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:37.184 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:02:37.184 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:37.185 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.186 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:02:37.186 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:37.186 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:02:37.187 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:02:37.188 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:02:37.189 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:02:37.198 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.198 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.198 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.198 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.198 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.199 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.199 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.200 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:37.206 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:37.211 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:37.212 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.213 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:37.213 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:02:37.213 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:02:37.215 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.218 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:02:37.218 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:02:37.236 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.237 INFO 4486 --- [ocp-schedules-7,6aed69fcc193ae87,fd39263f3ac750f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:02:37.241 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:02:37.244 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:02:37.246 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:02:37.253 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:02:37.254 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:02:37.257 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:02:37.264 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:02:37.274 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:02:37.281 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:37.282 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:37.283 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:37.289 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:02:37.309 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:02:37.318 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:02:37.320 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:02:37.324 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:02:37.328 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:02:37.334 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:02:37.338 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:37.341 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:02:37.342 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:37.352 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:02:37.352 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:37.367 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:02:37.368 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:02:37.371 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:02:37.378 INFO 4486 --- [host-check-host-status3,4abf702de83233a9,3d8e88568f5228f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:02:37.382 INFO 4486 --- [host-check-host-status4,4abf702de83233a9,1ae85e4bff148fc9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:02:37.391 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:02:37.401 INFO 4486 --- [host-check-host-status2,4abf702de83233a9,5e09df0b003e9627] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:02:37.413 INFO 4486 --- [ocp-async-8,0aeb25b40546b9b4,c55f7f999aa06181] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:37.429 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:02:37.434 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:37.436 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.437 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:02:37.439 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.440 INFO 4486 --- [host-check-host-status1,4abf702de83233a9,46bd1ef6e012fbc8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:02:37.440 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.441 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:02:37.452 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.452 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.452 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.452 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.452 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.453 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.480 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:02:37.528 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:02:37.549 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:37.572 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:37.589 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:37.590 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.591 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:02:37.593 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:02:37.598 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.599 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:02:37.600 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.601 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.602 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:02:37.610 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.610 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.610 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.610 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.610 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.611 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.618 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:02:37.836 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:02:37.850 INFO 4486 --- [ocp-schedules-27,e2864ee090d1f21a,da58909f2fafd5b3] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2024-10-16 14:02:37.850 INFO 4486 --- [ocp-schedules-27,e2864ee090d1f21a,da58909f2fafd5b3] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2024-10-16 14:02:37.881 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:37.883 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.884 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:02:37.885 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.886 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.887 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:02:37.898 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.898 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.898 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.898 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.898 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:37.898 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:37.919 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:02:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:02:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:02:37.994 INFO 4486 --- [prometheus-minute-schedule-1,c784dc16f63f6a48,37c4ae754ca2d883] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:38.000 INFO 4486 --- [prometheus-minute-schedule-1,c784dc16f63f6a48,37c4ae754ca2d883] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:38.056 INFO 4486 --- [host-check-clockdiff2,ac058eb9b23e97a5,61e99e045b934754] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058558 4 4 , error= 2024-10-16 14:02:38.112 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:02:38.154 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:38.157 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:38.157 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:02:38.159 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:38.159 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:38.161 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:02:38.171 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:38.171 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:38.171 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:38.171 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:38.171 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:02:38.172 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:38.236 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:02:38.426 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,80c1ad23f786bbdd,67abacf5a026da3f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=80c1ad23f786bbdd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:38.426 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,80c1ad23f786bbdd,67abacf5a026da3f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=80c1ad23f786bbdd, duration=1 ms] 2024-10-16 14:02:38.438 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:02:38.240484, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:02:38.439 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:02:38.240484, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:02:38.462 INFO 4486 --- [ocp-schedules-6,13ddf29aebb1f271,f549f1c853046ab8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 14:02:38.462 INFO 4486 --- [ocp-schedules-6,13ddf29aebb1f271,f549f1c853046ab8] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 14:02:38.463 INFO 4486 --- [ocp-schedules-6,13ddf29aebb1f271,f549f1c853046ab8] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 14:02:38.479 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:02:38.240484} } 2024-10-16 14:02:38.540 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:02:38.546 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:02:38.549 INFO 4486 --- [ocp-schedules-1,fdfc4dc308b8f7de,cf484161d7e86406] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:02:39.281 INFO 4486 --- [ocp-schedules-24,d0899c98e4156765,2171b84057c3456c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:02:39.413 INFO 4486 --- [ocp-async-4,205cc1d4f227dbb0,4756c51102f9097f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:40.006 INFO 4486 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:02:40.006 INFO 4486 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:02:40.026 INFO 4486 --- [Check obproxy2,cc50d2ea9f985d71,925a2e952e510ae4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:02:40.026 INFO 4486 --- [Check obproxy1,fe835428f5f9bd1e,c571bb3ebe84c0cc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:02:40.030 INFO 4486 --- [Check obproxy2,cc50d2ea9f985d71,925a2e952e510ae4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:40.030 INFO 4486 --- [Collect obproxy1,707a9b7da9b7fcb6,b81d454790b49b68] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:02:40.030 INFO 4486 --- [Collect obproxy2,eee0c8572fd5b1ae,0aae4709754f28a7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:02:40.030 INFO 4486 --- [Collect obproxy1,707a9b7da9b7fcb6,b81d454790b49b68] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:40.030 INFO 4486 --- [Collect obproxy2,eee0c8572fd5b1ae,0aae4709754f28a7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:40.032 INFO 4486 --- [Check obproxy1,fe835428f5f9bd1e,c571bb3ebe84c0cc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:02:40.053 INFO 4486 --- [Collect obproxy2,eee0c8572fd5b1ae,0aae4709754f28a7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:02:40.054 INFO 4486 --- [Check obproxy1,fe835428f5f9bd1e,c571bb3ebe84c0cc] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:02:40.054 INFO 4486 --- [Check obproxy1,fe835428f5f9bd1e,c571bb3ebe84c0cc] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:02:40.054 INFO 4486 --- [Check obproxy2,cc50d2ea9f985d71,925a2e952e510ae4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:02:40.054 INFO 4486 --- [Check obproxy2,cc50d2ea9f985d71,925a2e952e510ae4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:02:40.055 INFO 4486 --- [Collect obproxy1,707a9b7da9b7fcb6,b81d454790b49b68] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:02:40.058 INFO 4486 --- [Collect obproxy2,eee0c8572fd5b1ae,0aae4709754f28a7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:02:40.059 INFO 4486 --- [Collect obproxy1,707a9b7da9b7fcb6,b81d454790b49b68] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:02:40.062 INFO 4486 --- [Collect obproxy1,707a9b7da9b7fcb6,b81d454790b49b68] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:02:40.063 INFO 4486 --- [Collect obproxy2,eee0c8572fd5b1ae,0aae4709754f28a7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:02:40.457 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,62ab01de5ba2667c,cc2a4ff670579c29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=62ab01de5ba2667c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:40.457 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,62ab01de5ba2667c,cc2a4ff670579c29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=62ab01de5ba2667c, duration=1 ms] 2024-10-16 14:02:40.550 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:40.994 INFO 4486 --- [prometheus-minute-schedule-1,9e8259d966b27808,36cd343035e9efe7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:40.995 INFO 4486 --- [prometheus-hour-schedule-1,f6d4f8a300104b50,a3cb9d3576365ca3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:02:41.000 INFO 4486 --- [prometheus-minute-schedule-1,9e8259d966b27808,36cd343035e9efe7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:41.261 INFO 4486 --- [ocp-schedules-16,23fb5aca4b361d50,842ab988cfe1a65b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:02:41.261 INFO 4486 --- [ocp-schedules-16,68a97d413e4b7d73,718835f62296a590] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:02:41.264 INFO 4486 --- [ocp-schedules-16,68a97d413e4b7d73,718835f62296a590] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:02:41.300 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.300 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:41.304 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.304 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:41.328 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.329 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:41.329 INFO 4486 --- [ocp-schedules-25,f86ac3232cc5f00c,f9d0a1d382f1c7e9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:02:41.335 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.339 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.340 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:02:41.369 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.370 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.370 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:41.371 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:02:41.376 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.380 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:02:41.382 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:41.398 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.398 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:41.413 INFO 4486 --- [ocp-async-6,d139cfaff303812a,8379cbccc588e139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:41.415 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.424 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.425 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.425 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:41.426 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:02:41.430 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:02:41.434 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:41.436 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:41.437 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.438 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.439 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:02:41.460 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:02:41.465 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:02:41.465 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:02:41.490 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.492 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.492 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:02:41.496 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.499 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:02:41.503 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:41.528 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.528 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:41.530 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:02:41.541 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.565 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.565 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:02:41.574 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.574 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:41.575 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:02:41.575 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:02:41.596 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.596 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:41.621 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.621 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:02:41.645 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.645 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:02:41.672 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:02:41.698 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.699 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.700 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:02:41.705 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:02:41.710 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:02:41.714 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:02:41.716 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:02:41.739 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:02:41.890 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:02:41.916 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:02:41.918 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:02:41.919 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:02:41.952 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:02:41.952 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:02:41.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,e8abb68fe4fb0ef1,affc3e5920652a76] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e8abb68fe4fb0ef1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:41.996 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,e8abb68fe4fb0ef1,affc3e5920652a76] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e8abb68fe4fb0ef1, duration=1 ms] 2024-10-16 14:02:42.188 INFO 4486 --- [ocp-schedules-3,16901bb0b9214faa,716910fc2385b3e2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:02:42.291 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,adbca0503872f0a0,156f9fefd8da39a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=adbca0503872f0a0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:42.291 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,adbca0503872f0a0,156f9fefd8da39a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=adbca0503872f0a0, duration=1 ms] 2024-10-16 14:02:43.029 INFO 4486 --- [ocp-async-3,26d426ed18f8ec01,63d3c36d06da0bd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:02:43.102 INFO 4486 --- [ocp-schedules-17,b1aa6027719ef7f2,a8289b65946a7ebb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:02:43.132 WARN 4486 --- [scanner-13322753-1,b1aa6027719ef7f2,c0fcfa236e01d253] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:02:43.211 INFO 4486 --- [alarm-detect-19,b1aa6027719ef7f2,3e8d1189a28bc6be] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058535 ~ 1729058553 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058550, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058550, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058550=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:02:43.413 INFO 4486 --- [ocp-async-1,23a95cad5e101671,670e412b5cca55b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:43.551 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:43.994 INFO 4486 --- [prometheus-minute-schedule-1,70da8cd8587c428f,b156999f56a195ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:44.000 INFO 4486 --- [prometheus-minute-schedule-1,70da8cd8587c428f,b156999f56a195ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:45.145 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d8ad52a62e0da386,e55f061583134ecc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=d8ad52a62e0da386, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:02:45.146 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,d8ad52a62e0da386,e55f061583134ecc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=d8ad52a62e0da386, duration=2 ms] 2024-10-16 14:02:45.413 INFO 4486 --- [ocp-async-4,addccf177ae64119,242aa5384d332461] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:45.599 INFO 4486 --- [ocp-schedules-4,d021f57942df98d9,99af272b158dfe9b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:02:45.604 INFO 4486 --- [ocp-schedules-4,d021f57942df98d9,99af272b158dfe9b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:02:45.604 INFO 4486 --- [ocp-schedules-4,d021f57942df98d9,99af272b158dfe9b] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:02:45.637 INFO 4486 --- [ocp-schedules-17,1bfd96a6c1881902,463b4c83e69afe18] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:02:45.637 INFO 4486 --- [ocp-schedules-17,1bfd96a6c1881902,463b4c83e69afe18] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:02:46.552 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:46.994 INFO 4486 --- [prometheus-minute-schedule-1,d8f8f89bd9208ed3,dfb94f1588b44377] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:46.999 INFO 4486 --- [prometheus-minute-schedule-1,d8f8f89bd9208ed3,dfb94f1588b44377] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:47.070 INFO 4486 --- [ocp-schedules-19,faf0b87decbd50b5,e295c85ae9b7d753] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:02:47.161 INFO 4486 --- [ocp-schedules-14,555c8a863737c37a,a2d8ce0822a99e9d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:02:47.287 INFO 4486 --- [ocp-schedules-24,3de632ed177356bb,c42130f4cfc25544] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:02:47.414 INFO 4486 --- [ocp-async-2,d4e86dc476587b5a,e4eff9a90b8841a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:48.431 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,fcdcb62e77ba9540,8703aaf0bea26158] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fcdcb62e77ba9540, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:48.431 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,fcdcb62e77ba9540,8703aaf0bea26158] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fcdcb62e77ba9540, duration=1 ms] 2024-10-16 14:02:49.414 INFO 4486 --- [ocp-async-1,546322402ce901d8,3d525ac4e66da26b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:49.553 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:49.994 INFO 4486 --- [prometheus-minute-schedule-1,9edd0b9254647758,84addab946053bd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:50.000 INFO 4486 --- [prometheus-minute-schedule-1,9edd0b9254647758,84addab946053bd5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:50.462 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,25160477e03b0332,9ca4c902d67cdb03] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=25160477e03b0332, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:50.463 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,25160477e03b0332,9ca4c902d67cdb03] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=25160477e03b0332, duration=3 ms] 2024-10-16 14:02:50.569 INFO 4486 --- [ocp-schedules-29,cb798f73a0d01125,9b22b092b0a00946] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:02:50.569 INFO 4486 --- [ocp-schedules-29,cb798f73a0d01125,9b22b092b0a00946] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:02:50.575 INFO 4486 --- [ocp-schedules-29,cb798f73a0d01125,9b22b092b0a00946] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:02:50.575 INFO 4486 --- [ocp-schedules-29,cb798f73a0d01125,9b22b092b0a00946] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:02:50.578 INFO 4486 --- [ocp-schedules-29,cb798f73a0d01125,9b22b092b0a00946] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:02:50.578 INFO 4486 --- [ocp-schedules-29,cb798f73a0d01125,9b22b092b0a00946] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 14:02:51.335 INFO 4486 --- [ocp-schedules-24,dec5164ebfb7fa46,aa375ef19d602ffa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:02:51.414 INFO 4486 --- [ocp-async-9,b0f580ff46ffecaa,877312570b4fc874] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:51.564 INFO 4486 --- [ocp-schedules-10,ec3d853be2788c01,4cade9ae7a15b2ad] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:02:51.564 INFO 4486 --- [ocp-schedules-10,ec3d853be2788c01,4cade9ae7a15b2ad] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:02:51.564 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:02:51.564 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:02:51.564 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:02:51.564 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:02:51.564 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:02:51.570 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:02:51.570 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:02:51.570 INFO 4486 --- [ocp-schedules-10,ec3d853be2788c01,4cade9ae7a15b2ad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:02:51.570 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:02:51.570 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:02:51.572 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:02:51.572 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:02:51.572 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:02:51.572 INFO 4486 --- [ocp-schedules-10,ec3d853be2788c01,4cade9ae7a15b2ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:02:51.572 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:02:51.595 INFO 4486 --- [ocp-schedules-10,ec3d853be2788c01,4cade9ae7a15b2ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:02:51.595 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:02:51.595 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:02:51.598 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:02:51.598 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:02:51.601 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:02:51.603 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:02:51.624 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:02:51.637 INFO 4486 --- [ocp-schedules-10,ec3d853be2788c01,4cade9ae7a15b2ad] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:02:52.000 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,c7ef0c635d44fad4,164f3f96213e8f1f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c7ef0c635d44fad4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:52.000 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,c7ef0c635d44fad4,164f3f96213e8f1f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c7ef0c635d44fad4, duration=1 ms] 2024-10-16 14:02:52.295 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6f0bada37816cbb7,21a3383b7db02207] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6f0bada37816cbb7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:52.295 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6f0bada37816cbb7,21a3383b7db02207] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6f0bada37816cbb7, duration=1 ms] 2024-10-16 14:02:52.554 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:52.994 INFO 4486 --- [prometheus-minute-schedule-1,fe12aef343823ff8,614b4d66ff764bef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:53.001 INFO 4486 --- [prometheus-minute-schedule-1,fe12aef343823ff8,614b4d66ff764bef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:53.030 INFO 4486 --- [ocp-async-5,6493cf159ba81d52,0cbd7d216583b884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:02:53.318 INFO 4486 --- [ocp-schedules-11,8252a616b52caa89,c3224ea976d17e1b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:02:53.387 WARN 4486 --- [scanner-13323131-1,8252a616b52caa89,a36e0248ebd844c3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:02:53.417 INFO 4486 --- [ocp-async-6,7cc12963252fab89,5a42c82a97609b8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:55.418 INFO 4486 --- [ocp-async-10,878be84069083fcf,4f451e21de5c4e21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:55.555 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:55.994 INFO 4486 --- [prometheus-minute-schedule-1,10ed4e19802e387e,8452f775431b6b6f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:56.001 INFO 4486 --- [prometheus-minute-schedule-1,10ed4e19802e387e,8452f775431b6b6f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:56.369 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7f47a92dc8fb1b94,c8a40826b35ca759] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=7f47a92dc8fb1b94, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:02:56.370 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,7f47a92dc8fb1b94,c8a40826b35ca759] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=7f47a92dc8fb1b94, duration=3 ms] 2024-10-16 14:02:57.418 INFO 4486 --- [ocp-async-5,b0da11bde32bf9b3,03923be8688fb914] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:02:58.437 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b9d599c6d81f2965,efa60b00d3c03432] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b9d599c6d81f2965, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:02:58.437 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b9d599c6d81f2965,efa60b00d3c03432] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b9d599c6d81f2965, duration=2 ms] 2024-10-16 14:02:58.556 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:02:58.994 INFO 4486 --- [prometheus-minute-schedule-1,14d60eef3be438fe,68b57eb630dccccb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058580, interval=60 2024-10-16 14:02:59.002 INFO 4486 --- [prometheus-minute-schedule-1,14d60eef3be438fe,68b57eb630dccccb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:02:59.285 INFO 4486 --- [ocp-schedules-19,2250a0d9278d253b,4cc1875949626a44] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:02:59.418 INFO 4486 --- [ocp-async-3,ed84fc64295f9c91,551cdadcc41d802a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:00.468 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e81923faaf67a678,404017f17a8be9a6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e81923faaf67a678, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:00.468 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,e81923faaf67a678,404017f17a8be9a6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e81923faaf67a678, duration=1 ms] 2024-10-16 14:03:00.655 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 14:03:00.674 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:00.676 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:00.702 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:00.704 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:00.728 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 14:03:00.729 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 14:03:00.752 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 14:03:00.754 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 14:03:00.778 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:00.779 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:00.804 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:03:00.806 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:03:00.829 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:03:00.831 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:03:00.853 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 14:03:00.854 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 14:03:00.879 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:03:00.880 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:03:00.902 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 14:03:00.903 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 14:03:00.928 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:00.929 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:00.954 INFO 4486 --- [ocp-schedules-15,fcfaed6b3b3e3024,77dd3eb918c46188] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 14:03:01.340 INFO 4486 --- [ocp-schedules-25,ed24fe4a76a4c548,b3ee5cf3d721a3b4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:03:01.418 INFO 4486 --- [ocp-async-6,e31c84ed22540d69,88c7aa2ebe566d04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:01.557 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:01.994 INFO 4486 --- [prometheus-minute-schedule-1,65e35326fba67b29,33ce41c2b9c71739] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:02.001 INFO 4486 --- [prometheus-minute-schedule-1,65e35326fba67b29,33ce41c2b9c71739] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:02.006 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1ee9bbc9ebe72b60,e9005b1fb57371d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1ee9bbc9ebe72b60, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:02.006 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1ee9bbc9ebe72b60,e9005b1fb57371d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1ee9bbc9ebe72b60, duration=1 ms] 2024-10-16 14:03:02.300 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,05b62de9b1d462e1,1f89f6467f1c1562] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=05b62de9b1d462e1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:02.300 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,05b62de9b1d462e1,1f89f6467f1c1562] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=05b62de9b1d462e1, duration=1 ms] 2024-10-16 14:03:02.776 INFO 4486 --- [ocp-schedules-4,2714077e444f782c,90b286d06eeee682] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:03:02.776+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:03:02.779 INFO 4486 --- [ocp-schedules-4,2714077e444f782c,90b286d06eeee682] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:03:02.784 INFO 4486 --- [ocp-schedules-4,2714077e444f782c,90b286d06eeee682] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 14:03:02.796 INFO 4486 --- [ocp-schedules-4,2714077e444f782c,90b286d06eeee682] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:03:03.030 INFO 4486 --- [ocp-async-7,caa156bb30778e6a,241ed331de0046aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:03:03.418 INFO 4486 --- [ocp-async-4,a4295f456100edf5,21ba28fdc9998162] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:03.500 INFO 4486 --- [ocp-schedules-24,dfa1597652e041eb,348d18c491686d8d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:03:03.573 WARN 4486 --- [scanner-13323309-1,dfa1597652e041eb,db574592623a0064] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:03:03.646 INFO 4486 --- [alarm-detect-7,dfa1597652e041eb,76f1605d7f49343e] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058550 ~ 1729058573 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058565, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058565, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058565=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:03:04.557 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:04.994 INFO 4486 --- [prometheus-minute-schedule-1,8e5c7c89c9bd6c8f,9892d1a6a57fd6c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:05.001 INFO 4486 --- [prometheus-minute-schedule-1,8e5c7c89c9bd6c8f,9892d1a6a57fd6c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:05.418 INFO 4486 --- [ocp-async-2,5eb712a80c5a3e3b,6f32d90c7b7a8ffa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:07.066 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:03:07.066 INFO 4486 --- [ocp-schedules-9,3b21b7cf37c7edd0,22afeefb9d05ac5f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:03:07.068 INFO 4486 --- [ocp-schedules-9,3b21b7cf37c7edd0,22afeefb9d05ac5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:07.081 INFO 4486 --- [ocp-schedules-21,27f833fcba3e0ae8,cdf97256f54d3c11] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:03:07.083 INFO 4486 --- [ocp-schedules-21,27f833fcba3e0ae8,cdf97256f54d3c11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:03:07.089 INFO 4486 --- [ocp-schedules-11,b8b545bd336d7504,22c7ddd364f877ad] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:03:07.090 INFO 4486 --- [ocp-schedules-10,016e4a685aec5c1b,fce2e7d181166d2b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:03:07.090 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:03:07.090 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:03:07.091 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:03:07.091 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:03:07.091 INFO 4486 --- [ocp-schedules-29,3a27beb6d3ae9fcc,f76c99f6cf2081c0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:03:07.091 INFO 4486 --- [host-check-agent-status1,016e4a685aec5c1b,c334c9a8a77249b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:03:07.091 INFO 4486 --- [host-check-agent-status3,016e4a685aec5c1b,b8e8e650a0e5b9b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:03:07.092 INFO 4486 --- [host-check-agent-status4,016e4a685aec5c1b,a067126fe87d752c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:03:07.092 INFO 4486 --- [host-check-agent-status2,016e4a685aec5c1b,47f643c236a9861e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:03:07.092 INFO 4486 --- [host-check-clockdiff3,3a27beb6d3ae9fcc,7bd7f09e97e7ebbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:03:07.092 INFO 4486 --- [host-check-clockdiff2,3a27beb6d3ae9fcc,aa12a251bf390a41] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:03:07.092 INFO 4486 --- [host-check-clockdiff1,3a27beb6d3ae9fcc,81d52866bdbd9754] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:03:07.094 INFO 4486 --- [host-check-clockdiff4,3a27beb6d3ae9fcc,cd05ce0620b47538] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:03:07.098 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:07.098 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:07.098 INFO 4486 --- [host-check-agent-status3,016e4a685aec5c1b,b8e8e650a0e5b9b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:07.098 INFO 4486 --- [host-check-agent-status4,016e4a685aec5c1b,a067126fe87d752c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:07.099 INFO 4486 --- [host-check-agent-status2,016e4a685aec5c1b,47f643c236a9861e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:07.100 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:07.100 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:07.101 INFO 4486 --- [host-check-agent-status4,016e4a685aec5c1b,a067126fe87d752c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:07.101 INFO 4486 --- [host-check-agent-status3,016e4a685aec5c1b,b8e8e650a0e5b9b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:07.101 INFO 4486 --- [host-check-agent-status2,016e4a685aec5c1b,47f643c236a9861e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:07.101 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:07.101 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:07.102 INFO 4486 --- [host-check-agent-status1,016e4a685aec5c1b,c334c9a8a77249b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:07.122 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:07.126 INFO 4486 --- [host-check-agent-status1,016e4a685aec5c1b,c334c9a8a77249b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:07.126 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:07.127 INFO 4486 --- [host-check-clockdiff1,3a27beb6d3ae9fcc,81d52866bdbd9754] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058587 0 0 , error= 2024-10-16 14:03:07.130 INFO 4486 --- [host-check-clockdiff2,3a27beb6d3ae9fcc,aa12a251bf390a41] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058587 4 4 , error= 2024-10-16 14:03:07.139 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:07.140 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.141 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:03:07.156 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:07.158 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.158 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.159 INFO 4486 --- [host-check-agent-status3,016e4a685aec5c1b,b8e8e650a0e5b9b2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:03:07.159 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:03:07.159 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:07.160 INFO 4486 --- [host-check-agent-status1,016e4a685aec5c1b,c334c9a8a77249b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:03:07.161 INFO 4486 --- [host-check-agent-status2,016e4a685aec5c1b,47f643c236a9861e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:03:07.161 INFO 4486 --- [host-check-agent-status4,016e4a685aec5c1b,a067126fe87d752c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:03:07.162 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:03:07.162 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:03:07.162 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:03:07.163 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:03:07.166018699+08:00 2024-10-16 14:03:07.163 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:03:07.164 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:03:07.168071876+08:00 2024-10-16 14:03:07.164 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:03:07.163+08:00 2024-10-16 14:03:07.164 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:03:07.164158191+08:00 2024-10-16 14:03:07.165 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:03:07.164+08:00 2024-10-16 14:03:07.165 INFO 4486 --- [host-check-agent-status3,016e4a685aec5c1b,b8e8e650a0e5b9b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:03:07.165 INFO 4486 --- [host-check-agent-status3,016e4a685aec5c1b,b8e8e650a0e5b9b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:03:07.165 INFO 4486 --- [host-check-agent-status3,016e4a685aec5c1b,b8e8e650a0e5b9b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:03:07.165 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:03:07.164+08:00 2024-10-16 14:03:07.166 INFO 4486 --- [host-check-agent-status1,016e4a685aec5c1b,c334c9a8a77249b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:03:07.166 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:03:07.166 INFO 4486 --- [host-check-agent-status1,016e4a685aec5c1b,c334c9a8a77249b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:03:07.166 INFO 4486 --- [host-check-agent-status1,016e4a685aec5c1b,c334c9a8a77249b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:03:07.166 INFO 4486 --- [host-check-agent-status2,016e4a685aec5c1b,47f643c236a9861e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-agent-status2,016e4a685aec5c1b,47f643c236a9861e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-agent-status2,016e4a685aec5c1b,47f643c236a9861e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:03:07.170959755+08:00 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-agent-status4,016e4a685aec5c1b,a067126fe87d752c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-agent-status4,016e4a685aec5c1b,a067126fe87d752c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-agent-status4,016e4a685aec5c1b,a067126fe87d752c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:03:07.164+08:00 and status AVAILABLE 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:03:07.163+08:00 and status AVAILABLE 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:03:07.164+08:00 and status AVAILABLE 2024-10-16 14:03:07.167 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:03:07.167+08:00 2024-10-16 14:03:07.170 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:03:07.170 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:03:07.170 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.171 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:03:07.171 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:03:07.167+08:00 and status AVAILABLE 2024-10-16 14:03:07.171 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:03:07.173 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:03:07.179 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:07.180 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:07.181 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:07.181 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:07.181 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:07.181 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.182 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:07.182 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:03:07.183 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:07.183 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:07.191 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.192 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:03:07.195 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.196 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:03:07.202 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:07.202 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:07.205 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:07.206 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:07.209 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:07.210 INFO 4486 --- [ocp-schedules-25,08f38823fd026be6,150809b86089469e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:03:07.239 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:03:07.241 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:03:07.243 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:03:07.251 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:03:07.253 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:03:07.253 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:03:07.254 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:03:07.263 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:03:07.266 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:07.269 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:07.269 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:07.279 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:07.293 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:03:07.303 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:03:07.311 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:03:07.313 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:03:07.318 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:07.322 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:03:07.325 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:03:07.337 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:03:07.339 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:07.339 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:07.346 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:03:07.346 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:03:07.359 INFO 4486 --- [host-check-host-status3,b8b545bd336d7504,f5835743f9b58544] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:03:07.362 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:07.371 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:03:07.377 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:03:07.382 INFO 4486 --- [host-check-host-status4,b8b545bd336d7504,bb1a6f5267c3b316] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:03:07.388 INFO 4486 --- [host-check-host-status2,b8b545bd336d7504,575fd67fa41d3429] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:03:07.419 INFO 4486 --- [ocp-async-8,b92fa2cdaef1b073,4f1c55bf40790360] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:07.421 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:03:07.431 INFO 4486 --- [host-check-host-status1,b8b545bd336d7504,f7f733df50d4b50a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:03:07.552 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,43a4103ba39daf8b,6c52a7312fcd122a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=43a4103ba39daf8b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:03:07.552 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,43a4103ba39daf8b,6c52a7312fcd122a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=43a4103ba39daf8b, duration=2 ms] 2024-10-16 14:03:07.558 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:07.994 INFO 4486 --- [prometheus-minute-schedule-1,727bd3f1157df730,252020d801911a8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:07.999 INFO 4486 --- [prometheus-minute-schedule-1,727bd3f1157df730,252020d801911a8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:08.062 INFO 4486 --- [host-check-clockdiff4,3a27beb6d3ae9fcc,cd05ce0620b47538] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058588 4 4 , error= 2024-10-16 14:03:08.063 INFO 4486 --- [host-check-clockdiff3,3a27beb6d3ae9fcc,7bd7f09e97e7ebbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058588 5 5 , error= 2024-10-16 14:03:08.443 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b83e20412871206c,406f06ef23ac8672] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b83e20412871206c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:08.443 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b83e20412871206c,406f06ef23ac8672] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b83e20412871206c, duration=1 ms] 2024-10-16 14:03:09.251 INFO 4486 --- [metric-archive-minute-15,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=752 2024-10-16 14:03:09.251 INFO 4486 --- [metric-archive-minute-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=752 2024-10-16 14:03:09.256 INFO 4486 --- [metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=804 2024-10-16 14:03:09.256 INFO 4486 --- [metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=804 2024-10-16 14:03:09.275 ERROR 4486 --- [metric-archive-minute-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_minute error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_minute` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=835727) Duplicate entry '669741696490737664-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.274048] [YB420AC531B8-00060B7DC0C89CDC-0-0]; nested exception is java.sql.BatchUpdateException: (conn=835727) Duplicate entry '669741696490737664-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.274048] [YB420AC531B8-00060B7DC0C89CDC-0-0] 2024-10-16 14:03:09.298 ERROR 4486 --- [metric-archive-minute-15,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_minute error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_minute` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '586774768457945088-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.297348] [YB420AC531B8-00060B7DB4196CB6-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '586774768457945088-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.297348] [YB420AC531B8-00060B7DB4196CB6-0-0] 2024-10-16 14:03:09.371 ERROR 4486 --- [metric-archive-minute-12,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_minute error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_minute` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1694585) Duplicate entry '587073990944165888-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.370218] [YB420AC531B8-00060B7DB76886D2-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1694585) Duplicate entry '587073990944165888-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.370218] [YB420AC531B8-00060B7DB76886D2-0-0] 2024-10-16 14:03:09.385 ERROR 4486 --- [metric-archive-minute-21,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_minute error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_minute` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2457465) Duplicate entry '586774769061924864-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.383966] [YB420AC531B8-00060B7DA1ABF1ED-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2457465) Duplicate entry '586774769061924864-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.383966] [YB420AC531B8-00060B7DA1ABF1ED-0-0] 2024-10-16 14:03:09.395 ERROR 4486 --- [metric-archive-minute-29,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_minute error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_minute` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2734081) Duplicate entry '587073993980841984-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.394042] [YB420AC531B8-00060B7DB03B33B8-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2734081) Duplicate entry '587073993980841984-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.394042] [YB420AC531B8-00060B7DB03B33B8-0-0] 2024-10-16 14:03:09.419 INFO 4486 --- [ocp-async-10,0d449275784ea822,34b53a206821d6a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:09.422 ERROR 4486 --- [metric-archive-minute-30,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_minute error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_minute` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2949508) Duplicate entry '669741696482349056-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.421744] [YB420AC531B8-00060B7D84EAB42E-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2949508) Duplicate entry '669741696482349056-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.421744] [YB420AC531B8-00060B7D84EAB42E-0-0] 2024-10-16 14:03:09.424 ERROR 4486 --- [metric-archive-minute-9,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_minute error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_minute` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '669741699544190976-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.423326] [YB420AC531B8-00060B7DBF586BF0-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '669741699544190976-1729054800' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:09.423326] [YB420AC531B8-00060B7DBF586BF0-0-0] 2024-10-16 14:03:09.674 INFO 4486 --- [metric-archive-minute-3,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=468, affectRows=468 2024-10-16 14:03:09.674 INFO 4486 --- [metric-archive-minute-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=468, validCount=468 2024-10-16 14:03:10.008 INFO 4486 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:03:10.008 INFO 4486 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:03:10.034 INFO 4486 --- [Check obproxy1,169143d44d389420,256ed700af1f981d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:03:10.034 INFO 4486 --- [Check obproxy2,6e2b52569918402a,c57f5d94f3e7b153] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:03:10.034 INFO 4486 --- [Check obproxy1,169143d44d389420,256ed700af1f981d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:10.034 INFO 4486 --- [Check obproxy2,6e2b52569918402a,c57f5d94f3e7b153] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:10.039 INFO 4486 --- [Collect obproxy1,a5025e1f81dd8ec4,ff208acc1538ba0c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:03:10.039 INFO 4486 --- [Collect obproxy1,a5025e1f81dd8ec4,ff208acc1538ba0c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:10.042 INFO 4486 --- [Collect obproxy2,50dea11296e143b2,d5a392cb6d8ff9b9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:03:10.042 INFO 4486 --- [Collect obproxy2,50dea11296e143b2,d5a392cb6d8ff9b9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:10.060 INFO 4486 --- [Check obproxy2,6e2b52569918402a,c57f5d94f3e7b153] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:03:10.060 INFO 4486 --- [Check obproxy1,169143d44d389420,256ed700af1f981d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:03:10.060 INFO 4486 --- [Check obproxy2,6e2b52569918402a,c57f5d94f3e7b153] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:03:10.060 INFO 4486 --- [Check obproxy1,169143d44d389420,256ed700af1f981d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:03:10.067 INFO 4486 --- [Collect obproxy2,50dea11296e143b2,d5a392cb6d8ff9b9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:03:10.067 INFO 4486 --- [Collect obproxy1,a5025e1f81dd8ec4,ff208acc1538ba0c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:03:10.071 INFO 4486 --- [Collect obproxy1,a5025e1f81dd8ec4,ff208acc1538ba0c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:03:10.073 INFO 4486 --- [Collect obproxy2,50dea11296e143b2,d5a392cb6d8ff9b9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:03:10.074 INFO 4486 --- [Collect obproxy1,a5025e1f81dd8ec4,ff208acc1538ba0c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:03:10.078 INFO 4486 --- [Collect obproxy2,50dea11296e143b2,d5a392cb6d8ff9b9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:03:10.134 INFO 4486 --- [metric-archive-minute-15,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2024-10-16 14:03:10.134 INFO 4486 --- [metric-archive-minute-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=1000 2024-10-16 14:03:10.193 INFO 4486 --- [metric-archive-minute-12,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=987 2024-10-16 14:03:10.193 INFO 4486 --- [metric-archive-minute-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=987 2024-10-16 14:03:10.202 INFO 4486 --- [metric-archive-minute-29,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=990 2024-10-16 14:03:10.202 INFO 4486 --- [metric-archive-minute-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=990 2024-10-16 14:03:10.204 INFO 4486 --- [metric-archive-minute-21,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=983 2024-10-16 14:03:10.204 INFO 4486 --- [metric-archive-minute-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=983 2024-10-16 14:03:10.204 INFO 4486 --- [metric-archive-minute-9,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=977 2024-10-16 14:03:10.204 INFO 4486 --- [metric-archive-minute-9,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=977 2024-10-16 14:03:10.254 INFO 4486 --- [metric-archive-minute-30,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=976 2024-10-16 14:03:10.254 INFO 4486 --- [metric-archive-minute-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1729054800, nodeCount=1000, validCount=976 2024-10-16 14:03:10.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,2830b9786931b061,4e6548e58ee2451b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2830b9786931b061, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:10.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,2830b9786931b061,4e6548e58ee2451b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2830b9786931b061, duration=2 ms] 2024-10-16 14:03:10.559 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:10.980 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:03:10.994 INFO 4486 --- [prometheus-minute-schedule-1,c5438ba8fd0d9a24,6829da560e4f41b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:11.000 INFO 4486 --- [prometheus-minute-schedule-1,c5438ba8fd0d9a24,6829da560e4f41b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:11.008 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:03:11.345 INFO 4486 --- [ocp-schedules-3,4b04049acd7d651f,a4a25037b0eb6f15] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:03:11.419 INFO 4486 --- [ocp-async-5,f6cd99fc51159949,04bd9d12d2fe3e50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:11.953 INFO 4486 --- [ocp-schedules-16,68a97d413e4b7d73,718835f62296a590] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:03:11.953 INFO 4486 --- [ocp-schedules-16,6da7d07c77b2bdce,882ff07c79eb0d9e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:03:11.954 INFO 4486 --- [ocp-schedules-16,6da7d07c77b2bdce,882ff07c79eb0d9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:11.983 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:11.983 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:11.994 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:11.995 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:12.011 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.011 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:12.012 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,bd17ca8d19b7b57e,7d612651fc7f1f00] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bd17ca8d19b7b57e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:12.012 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,bd17ca8d19b7b57e,7d612651fc7f1f00] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bd17ca8d19b7b57e, duration=1 ms] 2024-10-16 14:03:12.014 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.015 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.015 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:03:12.044 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.046 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.046 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:12.047 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:03:12.047 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.050 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:03:12.051 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:12.059 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.060 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:12.070 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.078 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.079 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.079 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:12.080 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:03:12.084 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:03:12.088 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:12.090 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:12.110 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.111 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.112 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:03:12.112 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:03:12.151 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:03:12.151 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:03:12.159 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.161 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.162 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:12.167 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.172 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:03:12.176 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:12.191 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:03:12.202 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.203 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.204 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:03:12.204 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.204 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:12.215 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:03:12.215 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:03:12.234 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.234 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:12.261 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.261 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:12.289 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.289 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:12.304 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fe8fe5bde68fa736,9d18d524c4945727] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fe8fe5bde68fa736, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:12.304 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fe8fe5bde68fa736,9d18d524c4945727] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fe8fe5bde68fa736, duration=1 ms] 2024-10-16 14:03:12.315 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.315 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:12.341 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:12.366 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.368 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.369 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:12.373 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:03:12.378 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:03:12.382 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:03:12.384 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:03:12.405 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:03:12.543 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:03:12.573 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:12.575 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:12.576 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:03:12.604 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:03:12.604 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:03:13.030 INFO 4486 --- [ocp-async-4,e5585d95fa7bd1c2,0e71e59e1fb1f398] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:03:13.419 INFO 4486 --- [ocp-async-6,991aceec46598f9d,fb982d667fe601fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:13.560 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:13.738 INFO 4486 --- [ocp-schedules-30,6eda6bf17352a28a,49e29deccefe626e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:03:13.778 WARN 4486 --- [scanner-13324519-1,6eda6bf17352a28a,470fcbb85c0edae1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:03:13.805 INFO 4486 --- [alarm-detect-7,6eda6bf17352a28a,dbb2c12156dc3299] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058565 ~ 1729058583 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058580, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058580, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058580=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:03:13.994 INFO 4486 --- [prometheus-minute-schedule-1,fbd78744065e71b7,ef37049868517a6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:14.001 INFO 4486 --- [prometheus-minute-schedule-1,fbd78744065e71b7,ef37049868517a6b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:14.594 ERROR 4486 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '586774769061924864-1729058520' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:14.593733] [YB420AC531B8-00060B7DB4196CC9-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '586774769061924864-1729058520' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:14.593733] [YB420AC531B8-00060B7DB4196CC9-0-0] 2024-10-16 14:03:14.597 ERROR 4486 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=835727) Duplicate entry '586679390278586368-1729058520' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:14.596259] [YB420AC531B8-00060B7D84EAB434-0-0]; nested exception is java.sql.BatchUpdateException: (conn=835727) Duplicate entry '586679390278586368-1729058520' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:14.596259] [YB420AC531B8-00060B7D84EAB434-0-0] 2024-10-16 14:03:14.605 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058520, nodeCount=1000, affectRows=1000 2024-10-16 14:03:14.620 ERROR 4486 --- [metric-archive-second-18,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=3990815) Duplicate entry '587073990944165888-1729058520' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:14.605424] [YB420AC531B8-00060B7DB76886D5-0-0]; nested exception is java.sql.BatchUpdateException: (conn=3990815) Duplicate entry '587073990944165888-1729058520' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:03:14.605424] [YB420AC531B8-00060B7DB76886D5-0-0] 2024-10-16 14:03:14.625 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058520, nodeCount=1000, affectRows=1000 2024-10-16 14:03:14.625 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058520, nodeCount=1000, validCount=1000 2024-10-16 14:03:14.628 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058520, nodeCount=999, validCount=999 2024-10-16 14:03:14.653 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058520, nodeCount=88, affectRows=88 2024-10-16 14:03:14.654 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058520, nodeCount=86, validCount=86 2024-10-16 14:03:14.810 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058520, nodeCount=372, affectRows=372 2024-10-16 14:03:14.812 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058520, nodeCount=372, validCount=372 2024-10-16 14:03:15.154 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058520, nodeCount=989, affectRows=989 2024-10-16 14:03:15.158 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058520, nodeCount=989, validCount=989 2024-10-16 14:03:15.261 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058520, nodeCount=480, affectRows=480 2024-10-16 14:03:15.420 INFO 4486 --- [ocp-async-10,bf0ec5563e9794bc,c408808a3b0717a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:15.919 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058520, nodeCount=480, validCount=480 2024-10-16 14:03:16.561 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:16.994 INFO 4486 --- [prometheus-minute-schedule-1,995460829606772b,006948deac447d72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:17.000 INFO 4486 --- [prometheus-minute-schedule-1,995460829606772b,006948deac447d72] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:17.070 INFO 4486 --- [ocp-schedules-9,3b21b7cf37c7edd0,22afeefb9d05ac5f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:03:17.164 INFO 4486 --- [ocp-schedules-21,27f833fcba3e0ae8,cdf97256f54d3c11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:03:17.291 INFO 4486 --- [ocp-schedules-7,d5a7ab4cfdecb17f,678d3d439c826adb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:03:17.420 INFO 4486 --- [ocp-async-3,1d570dbc070f3562,431546401cc302be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:18.448 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,54eefee9b2024467,4b33efa4ce0c9cfd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=54eefee9b2024467, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:18.448 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,54eefee9b2024467,4b33efa4ce0c9cfd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=54eefee9b2024467, duration=2 ms] 2024-10-16 14:03:19.290 INFO 4486 --- [ocp-schedules-30,103cf8aa2344f75a,9e8ecb76dc8d44ff] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:03:19.420 INFO 4486 --- [ocp-async-7,1a07a96054af76aa,604296350d524435] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:19.561 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:19.994 INFO 4486 --- [prometheus-minute-schedule-1,df20abef6e7c578e,c0ace84f20b6af4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:20.000 INFO 4486 --- [prometheus-minute-schedule-1,df20abef6e7c578e,c0ace84f20b6af4f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:20.479 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2e0bfdee5c3683a6,6651f49c42f83a46] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2e0bfdee5c3683a6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:20.479 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,2e0bfdee5c3683a6,6651f49c42f83a46] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2e0bfdee5c3683a6, duration=1 ms] 2024-10-16 14:03:20.996 INFO 4486 --- [prometheus-cache-monitor1,88a5237ad050f0b5,96a34f2c34c23859] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13797,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8129} 2024-10-16 14:03:20.996 INFO 4486 --- [prometheus-cache-monitor1,88a5237ad050f0b5,96a34f2c34c23859] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:03:21.350 INFO 4486 --- [ocp-schedules-25,3eae6243c8bb732b,ad43b5f8c00dfd93] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:03:21.420 INFO 4486 --- [ocp-async-1,230d13b7656fa499,9ac605ae50a2f0d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:22.018 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,45854ff28773616e,50c479b37c3f834d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=45854ff28773616e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:22.018 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,45854ff28773616e,50c479b37c3f834d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=45854ff28773616e, duration=1 ms] 2024-10-16 14:03:22.308 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f9c2a63db64fad69,e9eaf3a2fc42edd0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f9c2a63db64fad69, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:22.308 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f9c2a63db64fad69,e9eaf3a2fc42edd0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f9c2a63db64fad69, duration=1 ms] 2024-10-16 14:03:22.562 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:22.994 INFO 4486 --- [prometheus-minute-schedule-1,cd4c2dc94c8587cd,1d44525b9e947e62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:23.000 INFO 4486 --- [prometheus-minute-schedule-1,cd4c2dc94c8587cd,1d44525b9e947e62] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:23.030 INFO 4486 --- [ocp-async-6,cd3a1d60e18946e1,649b4f38ef53a766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:03:23.420 INFO 4486 --- [ocp-async-5,df08228dceff69a5,926bcfe01d02482d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:23.861 INFO 4486 --- [ocp-schedules-22,998ab263c8363616,c1b61ac9c5387d35] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:03:23.883 WARN 4486 --- [scanner-13324704-1,998ab263c8363616,eea46ae6ef33e5fa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:03:24.006 INFO 4486 --- [ocp-schedules-10,f5b709a7bd1a9178,0cb2ea9cf069edb5] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:03:25.282 INFO 4486 --- [ocp-schedules-23,e20904147d53b378,d6b5de1846871bf2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:03:25.421 INFO 4486 --- [ocp-async-8,c41df7ee09e2f217,506bfa871bfb179c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:25.563 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:25.994 INFO 4486 --- [prometheus-minute-schedule-1,96386573823cad62,010da017a1db39c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:26.001 INFO 4486 --- [prometheus-minute-schedule-1,96386573823cad62,010da017a1db39c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:26.926 INFO 4486 --- [ocp-schedules-22,ff0e7065dfde7e1f,e8894a2e88999d88] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:03:26.926 INFO 4486 --- [ocp-schedules-22,ff0e7065dfde7e1f,e8894a2e88999d88] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:03:26.960 INFO 4486 --- [Collect all tenants major compactions3,ff0e7065dfde7e1f,c1e1fe909d86acf9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:03:26.960 INFO 4486 --- [Collect all tenants major compactions4,ff0e7065dfde7e1f,5781924f3de0ce55] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:03:26.960 INFO 4486 --- [Collect all tenants major compactions2,ff0e7065dfde7e1f,32ffdd2f13b2ffac] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:03:26.960 INFO 4486 --- [Collect all tenants major compactions1,ff0e7065dfde7e1f,b037975bdd04365d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:03:26.960 INFO 4486 --- [Collect all tenants major compactions5,ff0e7065dfde7e1f,54a30612c6a46bf8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:03:26.973 INFO 4486 --- [Collect all tenants major compactions1,ff0e7065dfde7e1f,b037975bdd04365d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:26.973 INFO 4486 --- [Collect all tenants major compactions2,ff0e7065dfde7e1f,32ffdd2f13b2ffac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:26.974 INFO 4486 --- [Collect all tenants major compactions2,ff0e7065dfde7e1f,32ffdd2f13b2ffac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:03:26.975 INFO 4486 --- [Collect all tenants major compactions1,ff0e7065dfde7e1f,b037975bdd04365d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:03:26.978 INFO 4486 --- [Collect all tenants major compactions3,ff0e7065dfde7e1f,c1e1fe909d86acf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:26.978 INFO 4486 --- [Collect all tenants major compactions1,ff0e7065dfde7e1f,b037975bdd04365d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:03:26.978 INFO 4486 --- [Collect all tenants major compactions2,ff0e7065dfde7e1f,32ffdd2f13b2ffac] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:03:26.979 INFO 4486 --- [Collect all tenants major compactions1,ff0e7065dfde7e1f,b037975bdd04365d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:03:26.979 INFO 4486 --- [Collect all tenants major compactions2,ff0e7065dfde7e1f,32ffdd2f13b2ffac] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:03:26.979 INFO 4486 --- [Collect all tenants major compactions3,ff0e7065dfde7e1f,c1e1fe909d86acf9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:03:26.983 INFO 4486 --- [Collect all tenants major compactions3,ff0e7065dfde7e1f,c1e1fe909d86acf9] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:03:26.983 INFO 4486 --- [Collect all tenants major compactions3,ff0e7065dfde7e1f,c1e1fe909d86acf9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:03:26.991 INFO 4486 --- [Collect all tenants major compactions4,ff0e7065dfde7e1f,5781924f3de0ce55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:26.992 INFO 4486 --- [Collect all tenants major compactions5,ff0e7065dfde7e1f,54a30612c6a46bf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:26.993 INFO 4486 --- [Collect all tenants major compactions4,ff0e7065dfde7e1f,5781924f3de0ce55] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:03:26.993 INFO 4486 --- [Collect all tenants major compactions5,ff0e7065dfde7e1f,54a30612c6a46bf8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:03:26.996 INFO 4486 --- [Collect all tenants major compactions4,ff0e7065dfde7e1f,5781924f3de0ce55] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:03:26.996 INFO 4486 --- [Collect all tenants major compactions4,ff0e7065dfde7e1f,5781924f3de0ce55] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:03:26.998 INFO 4486 --- [Collect all tenants major compactions5,ff0e7065dfde7e1f,54a30612c6a46bf8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:03:26.998 INFO 4486 --- [Collect all tenants major compactions5,ff0e7065dfde7e1f,54a30612c6a46bf8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:03:26.999 INFO 4486 --- [ocp-schedules-22,ff0e7065dfde7e1f,e8894a2e88999d88] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:03:27.421 INFO 4486 --- [ocp-async-9,d8a1841d1caf7f05,37c37cda8e301e4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:27.935 INFO 4486 --- [ocp-schedules-7,1adc6d59f0986ea2,09b7cf7e11b4277d] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:03:27.935+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:03:27.937 INFO 4486 --- [ocp-schedules-7,1adc6d59f0986ea2,09b7cf7e11b4277d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:03:27.937 INFO 4486 --- [ocp-schedules-7,1adc6d59f0986ea2,09b7cf7e11b4277d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:03:27.938 INFO 4486 --- [ocp-schedules-7,1adc6d59f0986ea2,09b7cf7e11b4277d] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:03:28.454 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b83381280972a292,41afdca80453a6bf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b83381280972a292, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:28.454 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,b83381280972a292,41afdca80453a6bf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b83381280972a292, duration=2 ms] 2024-10-16 14:03:28.564 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:28.994 INFO 4486 --- [prometheus-minute-schedule-1,3a393bb893cac9bc,23358280f908e75d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:28.999 INFO 4486 --- [prometheus-minute-schedule-1,3a393bb893cac9bc,23358280f908e75d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:29.421 INFO 4486 --- [ocp-async-10,0ceeb256e743fbcb,26aad2bd5dad6831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:29.543 INFO 4486 --- [ocp-schedules-2,5a1999ba80c093d5,7be90016820e570d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 14:03:30.103 INFO 4486 --- [prometheus-second-schedule-1,5dd256f6666d28ca,e062339868f1a7c9] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:03:30.486 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e851f48e5883f9c7,12bb7667fc5141d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e851f48e5883f9c7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:30.486 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,e851f48e5883f9c7,12bb7667fc5141d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e851f48e5883f9c7, duration=2 ms] 2024-10-16 14:03:31.355 INFO 4486 --- [ocp-schedules-3,e52eb8089a0de368,fc1cc63ac31fc847] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:03:31.421 INFO 4486 --- [ocp-async-5,777aac53603628dc,b1f811c68019a843] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:31.564 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:31.994 INFO 4486 --- [prometheus-minute-schedule-1,846b0957b32d861d,8bc4e5d97c2281ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:32.000 INFO 4486 --- [prometheus-minute-schedule-1,846b0957b32d861d,8bc4e5d97c2281ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:32.023 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c0c5012e8fe4b2cc,dc0fefc111ce43b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c0c5012e8fe4b2cc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:32.023 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c0c5012e8fe4b2cc,dc0fefc111ce43b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c0c5012e8fe4b2cc, duration=1 ms] 2024-10-16 14:03:32.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8ddba5361d47c452,dbb9318920664c0f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8ddba5361d47c452, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:32.314 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8ddba5361d47c452,dbb9318920664c0f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8ddba5361d47c452, duration=2 ms] 2024-10-16 14:03:33.031 INFO 4486 --- [ocp-async-3,46dc9f1adce89ab4,fc2a772fd8656cbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:03:33.422 INFO 4486 --- [ocp-async-6,164ac1063ebcd371,70a5a33c5dd8c3f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:33.584 INFO 4486 --- [ocp-schedules-6,47f1b94046343fb7,acee40c708fa5405] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:03:33.586 INFO 4486 --- [ocp-schedules-6,47f1b94046343fb7,acee40c708fa5405] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:33.587 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:03:33.587 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:03:33.587 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:03:34.081 INFO 4486 --- [ocp-schedules-20,961214c5331e6d1d,e1131ebdec14194e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:03:34.166 WARN 4486 --- [scanner-13325039-1,961214c5331e6d1d,f7bdeee1b73df604] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:03:34.242 INFO 4486 --- [alarm-detect-4,961214c5331e6d1d,d8ea53723fec9ce4] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058580 ~ 1729058604 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058595, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058595, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058595=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:03:34.250 INFO 4486 --- [alarm-detect-4,961214c5331e6d1d,d8ea53723fec9ce4] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 14:03:34.565 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:34.994 INFO 4486 --- [prometheus-minute-schedule-1,eb6e2a928bc2a8f9,9466bd8670a4eff5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:35.001 INFO 4486 --- [prometheus-minute-schedule-1,eb6e2a928bc2a8f9,9466bd8670a4eff5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:35.422 INFO 4486 --- [ocp-async-4,54e6919c0af5f482,a4ddf6c82777854b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:37.063 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:03:37.064 INFO 4486 --- [ocp-schedules-2,3605097bfed5753a,b86429c1c88c1a23] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:03:37.066 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:03:37.067 INFO 4486 --- [ocp-schedules-30,54c7d0fbd2261b2a,7e5d0f1e9596a66b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:03:37.068 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:03:37.069 INFO 4486 --- [ocp-schedules-30,54c7d0fbd2261b2a,7e5d0f1e9596a66b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:37.070 INFO 4486 --- [ocp-schedules-2,3605097bfed5753a,b86429c1c88c1a23] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:03:37.081 INFO 4486 --- [ocp-schedules-24,5ded4117c550644c,fbdec56dae8daf9e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:03:37.083 INFO 4486 --- [ocp-schedules-24,5ded4117c550644c,fbdec56dae8daf9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:37.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:03:37.089 INFO 4486 --- [ocp-schedules-11,254834a90a7317da,361c786eb0fd9602] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:03:37.089 INFO 4486 --- [ocp-schedules-8,2e5ad4cc57191342,ae1d3e3540ce4b3b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:03:37.090 INFO 4486 --- [ocp-schedules-9,3ab58b14dedcc3c4,c1570bd282a9d031] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:03:37.090 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:03:37.090 INFO 4486 --- [host-check-clockdiff2,2e5ad4cc57191342,4a7f6a5baf5151a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:03:37.090 INFO 4486 --- [host-check-clockdiff4,2e5ad4cc57191342,752396032905d40b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:03:37.091 INFO 4486 --- [host-check-clockdiff1,2e5ad4cc57191342,d245d9721e4477b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:03:37.091 INFO 4486 --- [host-check-clockdiff3,2e5ad4cc57191342,14556bde316e99ec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:03:37.091 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:03:37.092 INFO 4486 --- [host-check-agent-status1,3ab58b14dedcc3c4,df13ca659e4dda36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:03:37.092 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:03:37.092 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:03:37.092 INFO 4486 --- [host-check-agent-status2,3ab58b14dedcc3c4,422843e29baec657] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:03:37.093 INFO 4486 --- [host-check-agent-status3,3ab58b14dedcc3c4,b03e14e7f00ddb3b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:03:37.093 INFO 4486 --- [host-check-agent-status4,3ab58b14dedcc3c4,eb4a73b360ef5fa7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:03:37.099 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:37.099 INFO 4486 --- [host-check-agent-status1,3ab58b14dedcc3c4,df13ca659e4dda36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:37.107 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:37.108 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:37.109 INFO 4486 --- [host-check-agent-status3,3ab58b14dedcc3c4,b03e14e7f00ddb3b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:37.109 INFO 4486 --- [host-check-agent-status2,3ab58b14dedcc3c4,422843e29baec657] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:37.123 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:37.124 INFO 4486 --- [host-check-agent-status1,3ab58b14dedcc3c4,df13ca659e4dda36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:37.124 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:37.124 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:37.124 INFO 4486 --- [host-check-agent-status3,3ab58b14dedcc3c4,b03e14e7f00ddb3b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:37.124 INFO 4486 --- [host-check-agent-status4,3ab58b14dedcc3c4,eb4a73b360ef5fa7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:37.124 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:37.127 INFO 4486 --- [host-check-clockdiff1,2e5ad4cc57191342,d245d9721e4477b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058617 0 0 , error= 2024-10-16 14:03:37.127 INFO 4486 --- [host-check-clockdiff2,2e5ad4cc57191342,4a7f6a5baf5151a0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058617 4 4 , error= 2024-10-16 14:03:37.131 INFO 4486 --- [host-check-clockdiff3,2e5ad4cc57191342,14556bde316e99ec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058617 4 4 , error= 2024-10-16 14:03:37.131 INFO 4486 --- [host-check-agent-status2,3ab58b14dedcc3c4,422843e29baec657] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:37.134 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:37.134 INFO 4486 --- [host-check-agent-status4,3ab58b14dedcc3c4,eb4a73b360ef5fa7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:37.137 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.138 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:37.139 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:03:37.158 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.158 INFO 4486 --- [host-check-agent-status1,3ab58b14dedcc3c4,df13ca659e4dda36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:03:37.159 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:37.159 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:03:37.160 INFO 4486 --- [host-check-agent-status4,3ab58b14dedcc3c4,eb4a73b360ef5fa7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:03:37.160 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:03:37.160 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:37.160 INFO 4486 --- [host-check-agent-status2,3ab58b14dedcc3c4,422843e29baec657] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:03:37.162 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:03:37.162 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:03:37.165860041+08:00 2024-10-16 14:03:37.162 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.163 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:03:37.162+08:00 2024-10-16 14:03:37.163 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:03:37.166809896+08:00 2024-10-16 14:03:37.163 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:03:37.163+08:00 2024-10-16 14:03:37.163 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:03:37.164 INFO 4486 --- [host-check-agent-status3,3ab58b14dedcc3c4,b03e14e7f00ddb3b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:03:37.164 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:03:37.167947085+08:00 2024-10-16 14:03:37.164 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:03:37.165 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:03:37.164+08:00 2024-10-16 14:03:37.165 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:03:37.162+08:00 and status AVAILABLE 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status1,3ab58b14dedcc3c4,df13ca659e4dda36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status1,3ab58b14dedcc3c4,df13ca659e4dda36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status1,3ab58b14dedcc3c4,df13ca659e4dda36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:03:37.166 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.166 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status4,3ab58b14dedcc3c4,eb4a73b360ef5fa7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status4,3ab58b14dedcc3c4,eb4a73b360ef5fa7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status4,3ab58b14dedcc3c4,eb4a73b360ef5fa7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:03:37.163+08:00 and status AVAILABLE 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status2,3ab58b14dedcc3c4,422843e29baec657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status2,3ab58b14dedcc3c4,422843e29baec657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:03:37.166 INFO 4486 --- [host-check-agent-status2,3ab58b14dedcc3c4,422843e29baec657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:03:37.167 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.167 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:03:37.167 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:03:37.168 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:03:37.164+08:00 and status AVAILABLE 2024-10-16 14:03:37.168 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:03:37.168 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:03:37.167880719+08:00 2024-10-16 14:03:37.168 INFO 4486 --- [host-check-agent-status3,3ab58b14dedcc3c4,b03e14e7f00ddb3b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:03:37.168 INFO 4486 --- [host-check-agent-status3,3ab58b14dedcc3c4,b03e14e7f00ddb3b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:03:37.168 INFO 4486 --- [host-check-agent-status3,3ab58b14dedcc3c4,b03e14e7f00ddb3b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:03:37.168 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:03:37.168 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:03:37.168+08:00 2024-10-16 14:03:37.170 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:03:37.170 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:03:37.172 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:03:37.168+08:00 and status AVAILABLE 2024-10-16 14:03:37.175 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:03:37.176 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:37.177 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:37.177 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:37.178 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.178 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.178 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.178 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.178 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.178 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.178 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:37.179 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:37.179 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:03:37.179 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.180 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:37.185 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:37.187 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:37.188 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.189 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:03:37.200 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.201 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:03:37.201 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:37.202 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:03:37.202 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:37.203 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.203 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:37.204 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:03:37.211 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:03:37.213 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.214 INFO 4486 --- [ocp-schedules-4,d8d1f6796df218bd,fd2ea761aeeff640] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:03:37.232 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:03:37.238 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:03:37.243 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:03:37.244 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:03:37.249 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:03:37.256 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:03:37.262 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:37.269 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:37.274 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:37.274 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:03:37.287 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:03:37.291 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:03:37.301 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:03:37.302 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:03:37.313 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:03:37.317 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:37.320 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:03:37.325 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:03:37.330 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:03:37.342 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:37.346 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:37.349 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:03:37.355 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:03:37.361 INFO 4486 --- [host-check-host-status3,254834a90a7317da,7ab8b15b971459b4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:03:37.366 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:03:37.377 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:03:37.379 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:03:37.382 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:03:37.387 INFO 4486 --- [host-check-host-status4,254834a90a7317da,95effc8d8127c97e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:03:37.390 INFO 4486 --- [host-check-host-status2,254834a90a7317da,e0345f57e374e726] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:03:37.422 INFO 4486 --- [ocp-async-7,4cb9eefff6a06cd3,db7f6acc10397ec6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:37.429 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:37.431 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.432 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:03:37.433 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.434 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.435 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:03:37.436 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:03:37.443 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.444 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.444 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.444 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.444 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.444 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.446 INFO 4486 --- [host-check-host-status1,254834a90a7317da,72c531af2055b375] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:03:37.476 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:03:37.518 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:03:37.566 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:37.575 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:37.590 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:37.592 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.593 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:03:37.594 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:03:37.594 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.595 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:03:37.596 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.597 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.597 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:03:37.606 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.607 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.607 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.607 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.607 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.607 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.616 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:03:37.814 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:03:37.859 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:37.861 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.862 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:03:37.863 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.864 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.864 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:03:37.873 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.873 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.873 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.873 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.873 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:37.874 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:37.891 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:03:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:03:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:03:37.994 INFO 4486 --- [prometheus-minute-schedule-1,65361173c2620cd7,110f5271b2528a4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:38.001 INFO 4486 --- [prometheus-minute-schedule-1,65361173c2620cd7,110f5271b2528a4e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:38.057 INFO 4486 --- [host-check-clockdiff4,2e5ad4cc57191342,752396032905d40b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058618 4 4 , error= 2024-10-16 14:03:38.108 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:03:38.154 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:38.157 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:38.158 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:03:38.160 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:38.161 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:38.162 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:03:38.171 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:38.171 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:38.171 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:38.171 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:38.171 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:03:38.172 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:38.232 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:03:38.430 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:03:38.237208, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:03:38.430 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:03:38.237208, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:03:38.461 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,e50ebdac9740cc5b,d3d78f6eaa84fbef] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e50ebdac9740cc5b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:38.462 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,e50ebdac9740cc5b,d3d78f6eaa84fbef] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e50ebdac9740cc5b, duration=3 ms] 2024-10-16 14:03:38.468 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:03:38.237208} } 2024-10-16 14:03:38.481 INFO 4486 --- [ocp-schedules-7,277c18f90c179337,9c6a0c188b67593f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13797,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8129} 2024-10-16 14:03:38.481 INFO 4486 --- [ocp-schedules-7,277c18f90c179337,9c6a0c188b67593f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:03:38.523 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:03:38.531 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:03:38.533 INFO 4486 --- [ocp-schedules-13,25cbe26a6f2f110f,23643736ae46c276] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:03:39.042 INFO 4486 --- [ocp-schedules-21,1950c2a3dec2ceae,87c0658179fdd01e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 14:03:39.044 INFO 4486 --- [ocp-schedules-21,1950c2a3dec2ceae,87c0658179fdd01e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:39.294 INFO 4486 --- [ocp-schedules-8,7ec5d808b3e25091,91ad000b1af9b3c2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:03:39.422 INFO 4486 --- [ocp-async-6,4e5ecd4d3841df5f,6c90f0b6ed3ee3d5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:39.786 INFO 4486 --- [ocp-schedules-7,7bb4443a1ff030db,8850d7fe696ddca3] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2024-10-16 14:03:39.788 INFO 4486 --- [ocp-schedules-7,7bb4443a1ff030db,8850d7fe696ddca3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:39.820 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:39.822 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:39.827 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:42:50.879398, 2024-10-16 14:03:39.797, 1000] 2024-10-16 14:03:39.827 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:03:03.102283, 2024-10-16 14:03:39.797, 1000] 2024-10-16 14:03:39.829 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2024-10-16 02:03:03.102283, endTime 2024-10-16 14:03:39.797 2024-10-16 14:03:39.832 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 5, startTime 2024-10-16 08:42:50.879398, endTime 2024-10-16 14:03:39.797 2024-10-16 14:03:39.833 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:39.837 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 02:06:51.05999, 2024-10-16 14:03:39.797, 1000] 2024-10-16 14:03:39.844 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1000003, startTime 2024-10-16 02:06:51.05999, endTime 2024-10-16 14:03:39.797 2024-10-16 14:03:39.854 INFO 4486 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:39.859 INFO 4486 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:39.860 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:39.860 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:39.874 INFO 4486 --- [Sync system event3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:39.875 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:39.888 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 12:59:45.930954, 2024-10-16 14:03:39.844, 1000] 2024-10-16 14:03:39.891 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 1 server events for cluster 5, startTime 2024-10-16 12:59:45.930954, endTime 2024-10-16 14:03:39.844, limit 1000, offset 0 2024-10-16 14:03:39.893 INFO 4486 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-16 08:00:14.755425, 2024-10-16 14:03:39.844, 1000] 2024-10-16 14:03:39.894 INFO 4486 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2024-10-16 08:00:14.755425, endTime 2024-10-16 14:03:39.844 2024-10-16 14:03:39.900 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 1 server events to database for cluster 5 2024-10-16 14:03:39.904 INFO 4486 --- [Sync system event3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2024-10-12 15:26:31.336743, 2024-10-16 14:03:39.847, 1000] 2024-10-16 14:03:39.910 INFO 4486 --- [Sync system event3,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1000003, startTime 2024-10-12 15:26:31.336743, endTime 2024-10-16 14:03:39.847 2024-10-16 14:03:39.914 INFO 4486 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:39.915 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:39.944 INFO 4486 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+QUERY_TIMEOUT(60000000) */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2024-10-16 12:59:45.930954, 2024-10-16 14:03:39.844, 1000, 1000] 2024-10-16 14:03:39.947 INFO 4486 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 5, startTime 2024-10-16 12:59:45.930954, endTime 2024-10-16 14:03:39.844 2024-10-16 14:03:40.006 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:03:40.007 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:03:40.026 INFO 4486 --- [Check obproxy2,0e08d962a56b5b39,eda7b06f13bfe6fe] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:03:40.026 INFO 4486 --- [Check obproxy1,87d5bc076eedf368,49f49efe581a22b8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:03:40.026 INFO 4486 --- [Check obproxy2,0e08d962a56b5b39,eda7b06f13bfe6fe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:40.026 INFO 4486 --- [Check obproxy1,87d5bc076eedf368,49f49efe581a22b8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:40.032 INFO 4486 --- [Collect obproxy2,381c630970c4703e,e80e0ae743c7826d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:03:40.032 INFO 4486 --- [Collect obproxy1,7af5b6e01ad8e19f,83aaffd9fab98bcd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:03:40.032 INFO 4486 --- [Collect obproxy1,7af5b6e01ad8e19f,83aaffd9fab98bcd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:40.032 INFO 4486 --- [Collect obproxy2,381c630970c4703e,e80e0ae743c7826d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:03:40.050 INFO 4486 --- [Check obproxy2,0e08d962a56b5b39,eda7b06f13bfe6fe] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:03:40.050 INFO 4486 --- [Check obproxy2,0e08d962a56b5b39,eda7b06f13bfe6fe] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:03:40.053 INFO 4486 --- [Collect obproxy2,381c630970c4703e,e80e0ae743c7826d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:03:40.056 INFO 4486 --- [Check obproxy1,87d5bc076eedf368,49f49efe581a22b8] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:03:40.056 INFO 4486 --- [Check obproxy1,87d5bc076eedf368,49f49efe581a22b8] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:03:40.059 INFO 4486 --- [Collect obproxy2,381c630970c4703e,e80e0ae743c7826d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:03:40.061 INFO 4486 --- [Collect obproxy1,7af5b6e01ad8e19f,83aaffd9fab98bcd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:03:40.063 INFO 4486 --- [Collect obproxy2,381c630970c4703e,e80e0ae743c7826d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:03:40.064 INFO 4486 --- [Collect obproxy1,7af5b6e01ad8e19f,83aaffd9fab98bcd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:03:40.066 INFO 4486 --- [Collect obproxy1,7af5b6e01ad8e19f,83aaffd9fab98bcd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:03:40.135 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2747aaf1da80423f,c785c0339157f23f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=2747aaf1da80423f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:03:40.135 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2747aaf1da80423f,c785c0339157f23f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=2747aaf1da80423f, duration=1 ms] 2024-10-16 14:03:40.492 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,edc34b89dece9fbc,ce942aa2c6f05553] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=edc34b89dece9fbc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:40.492 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,edc34b89dece9fbc,ce942aa2c6f05553] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=edc34b89dece9fbc, duration=1 ms] 2024-10-16 14:03:40.567 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:40.994 INFO 4486 --- [prometheus-minute-schedule-1,f43123be1630bd10,80caefd2afebd748] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:40.995 INFO 4486 --- [prometheus-hour-schedule-1,d9326708fd95ff53,56ce704c54ae9c7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:03:41.001 INFO 4486 --- [prometheus-minute-schedule-1,f43123be1630bd10,80caefd2afebd748] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:41.362 INFO 4486 --- [ocp-schedules-26,153abea6ae84efa7,e23633e2e0619870] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:03:41.422 INFO 4486 --- [ocp-async-4,50ae36a7b291abd2,cd4eb82f5539206e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:42.027 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,579ba4bb6ebe7965,baee3b259485bfb7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=579ba4bb6ebe7965, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:42.027 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,579ba4bb6ebe7965,baee3b259485bfb7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=579ba4bb6ebe7965, duration=1 ms] 2024-10-16 14:03:42.191 INFO 4486 --- [ocp-schedules-23,1483b1382b098a07,0621f690f16a95ad] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:03:42.311 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,806d0fc8caac3c22,36a4318ee16a1618] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=806d0fc8caac3c22, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:03:42.311 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,806d0fc8caac3c22,36a4318ee16a1618] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=806d0fc8caac3c22, duration=1 ms] 2024-10-16 14:03:42.318 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,285122eebc4e4c60,319a34b3938fb051] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=285122eebc4e4c60, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:42.318 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,285122eebc4e4c60,319a34b3938fb051] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=285122eebc4e4c60, duration=0 ms] 2024-10-16 14:03:42.604 INFO 4486 --- [ocp-schedules-16,6da7d07c77b2bdce,882ff07c79eb0d9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:03:42.604 INFO 4486 --- [ocp-schedules-16,a12a9a529cd43533,7762ac74c2c94218] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:03:42.606 INFO 4486 --- [ocp-schedules-16,a12a9a529cd43533,7762ac74c2c94218] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:03:42.633 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.634 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:42.644 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.645 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:42.666 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:42.667 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.667 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:03:42.673 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.674 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:42.700 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:42.701 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.702 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:42.703 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:03:42.703 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.706 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:03:42.708 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:42.718 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:42.718 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:42.729 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:42.738 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:42.739 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.740 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:42.741 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:03:42.745 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:03:42.750 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:42.752 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:42.780 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:03:42.800 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:42.802 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.803 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:03:42.812 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:03:42.812 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:03:42.854 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:42.856 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.857 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:03:42.860 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.864 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:03:42.867 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:42.869 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:03:42.878 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:42.879 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:42.880 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:03:42.890 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:03:42.890 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:03:42.894 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:42.894 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:42.922 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:42.922 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:42.947 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:42.948 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:42.977 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:42.977 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:03:43.004 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:43.004 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:03:43.031 INFO 4486 --- [ocp-async-10,aa9a9254dce53f91,6a0c2ebb81d7e28d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:03:43.031 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:03:43.060 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:43.062 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:43.063 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:03:43.067 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:03:43.072 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:03:43.076 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:03:43.077 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:03:43.100 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:03:43.246 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:03:43.272 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:03:43.274 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:43.275 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:03:43.302 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:03:43.302 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058028393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:03:43.423 INFO 4486 --- [ocp-async-8,60ec74f0b1ac83f1,35dbdf2670567b67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:43.568 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:43.587 INFO 4486 --- [ocp-schedules-6,47f1b94046343fb7,acee40c708fa5405] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:03:43.994 INFO 4486 --- [prometheus-minute-schedule-1,5665066a41055c20,aa70ec14636efbb5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:44.000 INFO 4486 --- [prometheus-minute-schedule-1,5665066a41055c20,aa70ec14636efbb5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:44.333 INFO 4486 --- [ocp-schedules-1,5c2353c211dfebb4,0fcaf440d6f43240] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:03:44.382 WARN 4486 --- [scanner-13325301-1,5c2353c211dfebb4,e8b93d55bc239735] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:03:44.465 INFO 4486 --- [alarm-detect-7,5c2353c211dfebb4,eff1823854f06314] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058595 ~ 1729058614 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058610, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058610, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058610=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:03:45.423 INFO 4486 --- [ocp-async-1,4d7063ba359ee424,7690a679d675235d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:45.604 INFO 4486 --- [ocp-schedules-5,0a43fe7ec317d4ff,8821df03eb6a27cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:03:45.609 INFO 4486 --- [ocp-schedules-5,0a43fe7ec317d4ff,8821df03eb6a27cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:03:45.609 INFO 4486 --- [ocp-schedules-5,0a43fe7ec317d4ff,8821df03eb6a27cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:03:45.642 INFO 4486 --- [ocp-schedules-1,4260c0f1ccba36e2,43982ffdbe1f01f3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:03:45.642 INFO 4486 --- [ocp-schedules-1,4260c0f1ccba36e2,43982ffdbe1f01f3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:03:46.569 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:46.994 INFO 4486 --- [prometheus-minute-schedule-1,bf231fdbc4926bf2,b03841d24d1ad10d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:47.000 INFO 4486 --- [prometheus-minute-schedule-1,bf231fdbc4926bf2,b03841d24d1ad10d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:47.071 INFO 4486 --- [ocp-schedules-30,54c7d0fbd2261b2a,7e5d0f1e9596a66b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:03:47.164 INFO 4486 --- [ocp-schedules-24,5ded4117c550644c,fbdec56dae8daf9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:03:47.295 INFO 4486 --- [ocp-schedules-26,4845e413577df21a,4abf9004520fd478] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:03:47.423 INFO 4486 --- [ocp-async-3,c8aded11c6807004,9f4e647583b70f18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:48.467 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,eb2a3b9a62b30237,30d8c79e2a4d92f4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eb2a3b9a62b30237, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:48.467 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,eb2a3b9a62b30237,30d8c79e2a4d92f4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eb2a3b9a62b30237, duration=1 ms] 2024-10-16 14:03:48.786 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:48.787 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:03:48.788 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:03:48.788 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:03:48.789 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 14:03:48.963 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9bd7b020f3ca37f8,89e10514e827d9a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=9bd7b020f3ca37f8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:03:48.963 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,9bd7b020f3ca37f8,89e10514e827d9a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=9bd7b020f3ca37f8, duration=1 ms] 2024-10-16 14:03:49.073 INFO 4486 --- [ocp-schedules-21,1950c2a3dec2ceae,87c0658179fdd01e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 14:03:49.423 INFO 4486 --- [ocp-async-7,45e40d46da404935,625ebad27bc4074b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:49.569 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:49.947 INFO 4486 --- [ocp-schedules-7,7bb4443a1ff030db,8850d7fe696ddca3] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=3, successCount=3 2024-10-16 14:03:49.994 INFO 4486 --- [prometheus-minute-schedule-1,588c6a041b9c1aea,9add64a744af58ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:49.999 INFO 4486 --- [prometheus-minute-schedule-1,588c6a041b9c1aea,9add64a744af58ba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:50.498 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,62fadb8099598ef8,58f603dd6829f64e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=62fadb8099598ef8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:50.498 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,62fadb8099598ef8,58f603dd6829f64e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=62fadb8099598ef8, duration=1 ms] 2024-10-16 14:03:50.582 INFO 4486 --- [ocp-schedules-19,4362860c28d6921a,8b66dc16265577e0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:03:50.582 INFO 4486 --- [ocp-schedules-19,4362860c28d6921a,8b66dc16265577e0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:03:50.584 INFO 4486 --- [ocp-schedules-19,4362860c28d6921a,8b66dc16265577e0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:03:50.584 INFO 4486 --- [ocp-schedules-19,4362860c28d6921a,8b66dc16265577e0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:03:50.585 INFO 4486 --- [ocp-schedules-19,4362860c28d6921a,8b66dc16265577e0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:03:50.585 INFO 4486 --- [ocp-schedules-19,4362860c28d6921a,8b66dc16265577e0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 14:03:50.893 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 14:03:50.893 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 14:03:50.906 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 14:03:50.906 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 14:03:50.906 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 14:03:50.920 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 14:03:50.920 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 14:03:50.920 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 14:03:50.927 INFO 4486 --- [ocp-schedules-8,4655fb5e394c533d,ccb6eaee17391793] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 14:03:51.367 INFO 4486 --- [ocp-schedules-10,bdd2c3a1c6c00c61,e281b0e7827e275e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:03:51.424 INFO 4486 --- [ocp-async-6,0c047ef3ca4b661e,c67112a6c4a229ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:51.644 INFO 4486 --- [ocp-schedules-26,f0d053ed48c6af85,cbbdea7823dd2df0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:03:51.644 INFO 4486 --- [ocp-schedules-26,f0d053ed48c6af85,cbbdea7823dd2df0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:03:51.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:03:51.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:03:51.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:03:51.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:03:51.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:03:51.649 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:03:51.649 INFO 4486 --- [ocp-schedules-26,f0d053ed48c6af85,cbbdea7823dd2df0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:03:51.649 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:03:51.649 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:03:51.650 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:03:51.650 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:03:51.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:03:51.651 INFO 4486 --- [ocp-schedules-26,f0d053ed48c6af85,cbbdea7823dd2df0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:03:51.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:03:51.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:03:51.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:03:51.651 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:03:51.675 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:03:51.675 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:03:51.675 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:03:51.676 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:03:51.677 INFO 4486 --- [ocp-schedules-26,f0d053ed48c6af85,cbbdea7823dd2df0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:03:51.679 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:03:51.682 INFO 4486 --- [ocp-schedules-26,f0d053ed48c6af85,cbbdea7823dd2df0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:03:52.031 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a8af2d7539f6eed2,8a2f73fb4d3f7cdf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a8af2d7539f6eed2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:52.032 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,a8af2d7539f6eed2,8a2f73fb4d3f7cdf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a8af2d7539f6eed2, duration=1 ms] 2024-10-16 14:03:52.322 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0ac61e4a8fa1023f,a5b985d96b413f9c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0ac61e4a8fa1023f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:52.322 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0ac61e4a8fa1023f,a5b985d96b413f9c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0ac61e4a8fa1023f, duration=1 ms] 2024-10-16 14:03:52.495 INFO 4486 --- [ocp-schedules-23,8a0a6c0cbbe267be,a84b41e860c4f374] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:03:52.570 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:52.994 INFO 4486 --- [prometheus-minute-schedule-1,2a08a456cd3680cf,790b352d79188dff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:53.001 INFO 4486 --- [prometheus-minute-schedule-1,2a08a456cd3680cf,790b352d79188dff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:53.031 INFO 4486 --- [ocp-async-8,456453b4d7dd545b,3a47041158952d63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:03:53.424 INFO 4486 --- [ocp-async-3,f0c7a886672da41e,4426739070adfbfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:54.550 INFO 4486 --- [ocp-schedules-1,338f4cf406de826a,caa32e95241d5cec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:03:54.588 WARN 4486 --- [scanner-13325598-1,338f4cf406de826a,de5ca220f316643c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:03:54.806 INFO 4486 --- [ocp-schedules-4,4f64831db10d4fe0,8587c350e79eac6d] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T14:03:54.806+08:00, tenantSize=5 2024-10-16 14:03:54.811 INFO 4486 --- [ocp-schedules-4,4f64831db10d4fe0,8587c350e79eac6d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729058634806000 2024-10-16 14:03:54.811 INFO 4486 --- [ocp-schedules-4,4f64831db10d4fe0,8587c350e79eac6d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729058334741000 2024-10-16 14:03:54.838 INFO 4486 --- [ocp-schedules-4,4f64831db10d4fe0,8587c350e79eac6d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 14:03:54.844 INFO 4486 --- [ocp-schedules-4,4f64831db10d4fe0,8587c350e79eac6d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729058634841000 2024-10-16 14:03:54.844 INFO 4486 --- [ocp-schedules-4,4f64831db10d4fe0,8587c350e79eac6d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729058334765000 2024-10-16 14:03:54.879 INFO 4486 --- [ocp-schedules-4,4f64831db10d4fe0,8587c350e79eac6d] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 14:03:55.424 INFO 4486 --- [ocp-async-7,d75e25db88be1b9f,829f815b734281bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:55.571 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:55.994 INFO 4486 --- [prometheus-minute-schedule-1,82324e8e3165851d,1f9fb6d2035e213b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:56.000 INFO 4486 --- [prometheus-minute-schedule-1,82324e8e3165851d,1f9fb6d2035e213b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:57.424 INFO 4486 --- [ocp-async-6,8ab35da5845e3417,55ff29518a68e925] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:03:58.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fb365eeb8726a8c9,6d67fe2350e23dab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fb365eeb8726a8c9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:03:58.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fb365eeb8726a8c9,6d67fe2350e23dab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fb365eeb8726a8c9, duration=2 ms] 2024-10-16 14:03:58.572 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:03:58.994 INFO 4486 --- [prometheus-minute-schedule-1,1da944da1a0bff5e,57e74dc8542ecda9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058640, interval=60 2024-10-16 14:03:59.000 INFO 4486 --- [prometheus-minute-schedule-1,1da944da1a0bff5e,57e74dc8542ecda9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:03:59.299 INFO 4486 --- [ocp-schedules-26,1677cc982342e504,3f825989f0adcbd6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:03:59.424 INFO 4486 --- [ocp-async-4,d73d74dbba9f251b,a5cad30e8966a98f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:00.502 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3164277e93aa4153,22dcfc2b188a3c9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3164277e93aa4153, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:00.502 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3164277e93aa4153,22dcfc2b188a3c9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3164277e93aa4153, duration=1 ms] 2024-10-16 14:04:01.372 INFO 4486 --- [ocp-schedules-8,d3986fa191e78c04,edb8f715931e7f68] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:04:01.425 INFO 4486 --- [ocp-async-2,82e543fa98ae39ae,9857f4dea3e60c9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:01.573 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:01.994 INFO 4486 --- [prometheus-minute-schedule-1,88e3ba7a1105da7d,068546652a54ce50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:01.998 INFO 4486 --- [prometheus-minute-schedule-1,88e3ba7a1105da7d,068546652a54ce50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:02.036 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,70072e873f1947e1,1d0b3aa0ddda7984] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=70072e873f1947e1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:02.036 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,70072e873f1947e1,1d0b3aa0ddda7984] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=70072e873f1947e1, duration=1 ms] 2024-10-16 14:04:02.333 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ab0ad904a7d3882a,96661309a682f28f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ab0ad904a7d3882a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:02.333 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ab0ad904a7d3882a,96661309a682f28f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ab0ad904a7d3882a, duration=1 ms] 2024-10-16 14:04:02.527 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:04:02.528 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=7 2024-10-16 14:04:02.528 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2024-10-16 14:04:02.528 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2024-10-16 14:04:02.528 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1000015 2024-10-16 14:04:02.552 WARN 4486 --- [ForkJoinPool.commonPool-worker-25,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 14:04:02.555 WARN 4486 --- [ForkJoinPool.commonPool-worker-31,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 14:04:02.567 WARN 4486 --- [ForkJoinPool.commonPool-worker-30,,] com.oceanbase.ocp.common.json.JsonUtils : failed to covert json string to class:class com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$ObCleanerConfig, json:${ocp.agent.ob-cleaner.config:{"ob_log":{"disk_threshold":80,"rules":[{"file_regex":"[a-z]+.[a-z]+.log.[0-9]+","retention_days":8,"keep_percentage":60},{"file_regex":"([a-z]+.)?[a-z]+.log.wf.[0-9]+","retention_days":30,"keep_percentage":80}]},"core_log":{"disk_threshold":80,"rules":[{"file_regex":"core.[0-9]+","retention_days":8,"keep_percentage":60}]}} 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_all_error_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.593 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.197.49.180 2024-10-16 14:04:02.594 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 7 agent sql collect property 2024-10-16 14:04:02.594 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:02.597 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:02.616 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 14:04:02.616 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 14:04:02.617 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.197.49.184 2024-10-16 14:04:02.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2024-10-16 14:04:02.618 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2024-10-16 14:04:02.632 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1000015 agent sql collect property 2024-10-16 14:04:02.632 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2024-10-16 14:04:02.632 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2024-10-16 14:04:02.635 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:02.635 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:02.638 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:02.638 INFO 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 7, do nothing 2024-10-16 14:04:02.639 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:02.644 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:02.648 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:02.675 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=root), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313637), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.184), (ob.run.path=/root/oceanbase), (ob.install.path=/root/oceanbase), (ob.data.path=/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=OceanBase), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLevels":["INFO"],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"ob_log_fail_check_localserver","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[SHARE.PT\\] init_batch_rpc_renew_struct.*fail to check is local ser 2024-10-16 14:04:02.683 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313639), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=10.197.49.178), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data), (monagent.cluster.id=5), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/redo), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=obce_3zones), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=2), (monagent.obproxy.cluster.name=obce_3zones), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=10.197.49.178), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition from"],"includeKeywordsRelation":"Or","level":3,"logLev 2024-10-16 14:04:02.688 ERROR 4486 --- [ocp-schedules-13,ade6930ccf0f260e,3d4221a539149efc] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:79) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$executeIfNotReadOnly$1(AbstractSchedules.java:133) at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor1649.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.oceanbase.ocp.server.common.scheduled.AbstractSchedules.lambda$scheduledTasks$0(AbstractSchedules.java:77) ... 10 common frames omitted Caused by: com.oceanbase.ocp.executor.exception.HttpRequestFailedException: [AgentClient]:http request is failed, response:Unexpected error: key monagent.ob.cgroup.status is not found in config properties! at com.oceanbase.ocp.executor.internal.template.HttpTemplate.checkSuccess(HttpTemplate.java:476) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.doPost(HttpTemplate.java:286) at com.oceanbase.ocp.executor.internal.template.HttpTemplate.post(HttpTemplate.java:114) at com.oceanbase.ocp.executor.executor.AgentExecutor.updateAgentConfig(AgentExecutor.java:193) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentProperties(HostAgentConfigManagerImpl.java:188) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl.refreshAgentConfig(HostAgentConfigManagerImpl.java:196) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$FastClassBySpringCGLIB$$798eb4cd.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.compute.agent.manager.HostAgentConfigManagerImpl$$EnhancerBySpringCGLIB$$eb50b66a.refreshAgentConfig() at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl.refreshHostAgentConfig(HostAgentServiceImpl.java:734) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$FastClassBySpringCGLIB$$e7947c56.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.compute.agent.HostAgentServiceImpl$$EnhancerBySpringCGLIB$$5f28aa5.refreshHostAgentConfig() at com.oceanbase.ocp.server.common.scheduled.HostSchedules.lambda$refreshAgentConfig$0(HostSchedules.java:92) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 14:04:02.690 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.197.49.184), (ob.sync.status=VALID), (monagent.ob.cluster.id=1701313644), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (monagent.host.ip=172.28.46.152), (ob.run.path=/home/admin/oceanbase), (ob.install.path=/home/admin/oceanbase), (ob.data.path=/data/1), (monagent.cluster.id=1000003), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/data/log1), (ob.logcleaner.enabled=true), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.cluster.name=glob_3zone), (ob.logtailer.enabled=true), (monagent.monitordb.user=root@ocp_monitor), (ob.logcleaner.run.internal=5m), (monagent.obproxy.cluster.id=1000002), (monagent.obproxy.cluster.name=xc_OB43), (monagent.obproxy.cluster.work.mode=configUrl), (monagent.host.ip=172.28.46.152), (monagent.obproxy.sql.port=2883), (obproxy.run.path=/home/admin/obproxy), (obproxy.install.path=/home/admin/obproxy), (obproxy.run.user=admin), (monagent.pipeline.obproxy.status=active), (monagent.obproxy.exporter.port=2884), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"ob_log_retry_remove_parts","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":["alarm"],"excludeKeywordsRelation":"Or","expression":{"metric":"observer_log"},"expressionType":"Log","includeKeywords":["\\[STORAGE\\] ob_partition_service.*retry remove partition fro 2024-10-16 14:04:02.696 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2024-10-16 14:04:02.699 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1000015, do nothing 2024-10-16 14:04:02.827 INFO 4486 --- [ocp-schedules-8,cb57be3f92604d8e,ce3b258f8870a3b5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:04:02.827+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:04:02.830 INFO 4486 --- [ocp-schedules-8,cb57be3f92604d8e,ce3b258f8870a3b5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:04:02.839 INFO 4486 --- [ocp-schedules-8,cb57be3f92604d8e,ce3b258f8870a3b5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:04:02.849 INFO 4486 --- [ocp-schedules-8,cb57be3f92604d8e,ce3b258f8870a3b5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:04:03.031 INFO 4486 --- [ocp-async-3,c3b7d4aa1f41b8ae,4434a702fc17d266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:04:03.425 INFO 4486 --- [ocp-async-9,e315058c75374c0c,94ea13e816ad3cfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:04.574 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:04.682 INFO 4486 --- [ocp-schedules-4,919b5f39075f1c91,1d526fdd79a0805a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:04:04.717 WARN 4486 --- [scanner-13325847-1,919b5f39075f1c91,64f3946030b5f624] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:04:04.783 INFO 4486 --- [alarm-detect-12,919b5f39075f1c91,1de185b689e528ed] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058610 ~ 1729058634 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058625, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058625, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058625=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:04:04.994 INFO 4486 --- [prometheus-minute-schedule-1,70abc325e4e6cbc0,03cce8afbe515962] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:05.000 INFO 4486 --- [prometheus-minute-schedule-1,70abc325e4e6cbc0,03cce8afbe515962] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:05.425 INFO 4486 --- [ocp-async-10,91da370fca10ed1e,6588dd8f4185ddd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:07.066 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:04:07.066 INFO 4486 --- [ocp-schedules-22,667bd633ade2ef0f,55edb4c30efb8179] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:04:07.068 INFO 4486 --- [ocp-schedules-22,667bd633ade2ef0f,55edb4c30efb8179] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:07.081 INFO 4486 --- [ocp-schedules-13,e9761a3c515565c9,82f59d30d67585ba] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:04:07.084 INFO 4486 --- [ocp-schedules-13,e9761a3c515565c9,82f59d30d67585ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:04:07.090 INFO 4486 --- [ocp-schedules-15,73ebf88d0329fc68,5a6af30473534fc1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:04:07.090 INFO 4486 --- [ocp-schedules-25,63db12627de6b2f4,ef480d4f59782a27] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:04:07.091 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:04:07.091 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:04:07.091 INFO 4486 --- [host-check-agent-status1,63db12627de6b2f4,1d73bac68a994395] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:04:07.091 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:04:07.091 INFO 4486 --- [ocp-schedules-3,e24bd0ebe896071f,1ad74e53384696f6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:04:07.091 INFO 4486 --- [host-check-agent-status2,63db12627de6b2f4,3b6f2f80e5c7b5bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:04:07.092 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:04:07.092 INFO 4486 --- [host-check-clockdiff1,e24bd0ebe896071f,97629b12b648b11f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:04:07.093 INFO 4486 --- [host-check-clockdiff2,e24bd0ebe896071f,daafa10f1e4bdc4c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:04:07.093 INFO 4486 --- [host-check-agent-status3,63db12627de6b2f4,9546aeb3fe9a3d52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:04:07.093 INFO 4486 --- [host-check-clockdiff3,e24bd0ebe896071f,8edb49f4a02c4921] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:04:07.093 INFO 4486 --- [host-check-agent-status4,63db12627de6b2f4,f31b06d4423941a2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:04:07.094 INFO 4486 --- [host-check-clockdiff4,e24bd0ebe896071f,b980f50f22a9dc05] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:04:07.099 INFO 4486 --- [host-check-agent-status1,63db12627de6b2f4,1d73bac68a994395] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:07.132 INFO 4486 --- [host-check-agent-status2,63db12627de6b2f4,3b6f2f80e5c7b5bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:07.133 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:07.133 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:07.148 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:07.148 INFO 4486 --- [host-check-clockdiff2,e24bd0ebe896071f,daafa10f1e4bdc4c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058647 4 4 , error= 2024-10-16 14:04:07.148 INFO 4486 --- [host-check-clockdiff1,e24bd0ebe896071f,97629b12b648b11f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058647 0 0 , error= 2024-10-16 14:04:07.148 INFO 4486 --- [host-check-clockdiff3,e24bd0ebe896071f,8edb49f4a02c4921] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058647 4 4 , error= 2024-10-16 14:04:07.148 INFO 4486 --- [host-check-agent-status1,63db12627de6b2f4,1d73bac68a994395] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:07.148 INFO 4486 --- [host-check-agent-status2,63db12627de6b2f4,3b6f2f80e5c7b5bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:07.148 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:07.152 INFO 4486 --- [host-check-agent-status3,63db12627de6b2f4,9546aeb3fe9a3d52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:07.156 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:07.157 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:07.157 INFO 4486 --- [host-check-agent-status4,63db12627de6b2f4,f31b06d4423941a2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:07.157 INFO 4486 --- [host-check-clockdiff4,e24bd0ebe896071f,b980f50f22a9dc05] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058647 4 4 , error= 2024-10-16 14:04:07.157 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:07.159 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:07.159 INFO 4486 --- [host-check-agent-status3,63db12627de6b2f4,9546aeb3fe9a3d52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:07.159 INFO 4486 --- [host-check-agent-status4,63db12627de6b2f4,f31b06d4423941a2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:07.165 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.165 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:07.166 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:04:07.186 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.186 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:04:07.187 INFO 4486 --- [host-check-agent-status1,63db12627de6b2f4,1d73bac68a994395] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:04:07.187 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:07.189 INFO 4486 --- [host-check-agent-status2,63db12627de6b2f4,3b6f2f80e5c7b5bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:04:07.189 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:04:07.193297042+08:00 2024-10-16 14:04:07.190 INFO 4486 --- [host-check-agent-status4,63db12627de6b2f4,f31b06d4423941a2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:04:07.190 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:04:07.190+08:00 2024-10-16 14:04:07.191 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:04:07.191 INFO 4486 --- [host-check-agent-status3,63db12627de6b2f4,9546aeb3fe9a3d52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-agent-status3,63db12627de6b2f4,9546aeb3fe9a3d52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-agent-status2,63db12627de6b2f4,3b6f2f80e5c7b5bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-agent-status2,63db12627de6b2f4,3b6f2f80e5c7b5bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-agent-status3,63db12627de6b2f4,9546aeb3fe9a3d52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-agent-status2,63db12627de6b2f4,3b6f2f80e5c7b5bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:04:07.195 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-agent-status1,63db12627de6b2f4,1d73bac68a994395] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:04:07.196 INFO 4486 --- [host-check-agent-status1,63db12627de6b2f4,1d73bac68a994395] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-agent-status3,63db12627de6b2f4,9546aeb3fe9a3d52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:04:07.195 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:04:07.196 INFO 4486 --- [host-check-agent-status1,63db12627de6b2f4,1d73bac68a994395] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:04:07.196 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:04:07.190+08:00 and status AVAILABLE 2024-10-16 14:04:07.197 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:04:07.196879926+08:00 2024-10-16 14:04:07.197 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:04:07.200978539+08:00 2024-10-16 14:04:07.197 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:04:07.198 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:04:07.197+08:00 2024-10-16 14:04:07.198 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:04:07.197+08:00 2024-10-16 14:04:07.198 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:04:07.200 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:04:07.203049396+08:00 2024-10-16 14:04:07.200 INFO 4486 --- [host-check-agent-status4,63db12627de6b2f4,f31b06d4423941a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:04:07.200 INFO 4486 --- [host-check-agent-status4,63db12627de6b2f4,f31b06d4423941a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:04:07.200 INFO 4486 --- [host-check-agent-status4,63db12627de6b2f4,f31b06d4423941a2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:04:07.200 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:04:07.200+08:00 2024-10-16 14:04:07.201 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:04:07.197+08:00 and status AVAILABLE 2024-10-16 14:04:07.201 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:04:07.197+08:00 and status AVAILABLE 2024-10-16 14:04:07.203 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:04:07.200+08:00 and status AVAILABLE 2024-10-16 14:04:07.203 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:04:07.204 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:04:07.205 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:04:07.208 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:07.209 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:07.210 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.212 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:07.213 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:07.214 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:07.214 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:07.214 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:07.215 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:04:07.216 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:07.223 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.224 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:07.233 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:07.234 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.235 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:07.237 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:07.238 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:07.239 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:07.246 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.247 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:07.249 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.252 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:04:07.262 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:07.263 INFO 4486 --- [ocp-schedules-11,c65f87d2f5640061,78f0871e666ec32b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:04:07.266 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:04:07.269 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:04:07.278 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:04:07.279 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:04:07.289 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:04:07.296 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:07.298 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:07.300 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:04:07.300 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:04:07.310 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:04:07.319 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:07.323 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:04:07.328 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:07.332 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:04:07.334 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:04:07.343 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:04:07.350 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:07.357 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:04:07.362 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:07.370 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:04:07.378 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:04:07.384 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:04:07.389 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:07.390 INFO 4486 --- [host-check-host-status3,73ebf88d0329fc68,36e39d6d41aa2ab1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:04:07.393 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:04:07.398 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:04:07.408 INFO 4486 --- [host-check-host-status4,73ebf88d0329fc68,f1ba9c5172a58d64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:04:07.417 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:07.425 INFO 4486 --- [ocp-async-3,524a98f951118e72,22ad43813b6a6858] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:07.432 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:04:07.443 INFO 4486 --- [host-check-host-status2,73ebf88d0329fc68,ddbd714c5b1f1ea9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:04:07.474 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:04:07.488 INFO 4486 --- [host-check-host-status1,73ebf88d0329fc68,25b7f1497a256be1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:04:07.574 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:07.994 INFO 4486 --- [prometheus-minute-schedule-1,0e04a02daae75050,746a3ee0d934c969] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:08.000 INFO 4486 --- [prometheus-minute-schedule-1,0e04a02daae75050,746a3ee0d934c969] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:08.478 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,b80ee2ecec90f429,f6a83ab331c6a7c5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b80ee2ecec90f429, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:08.478 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,b80ee2ecec90f429,f6a83ab331c6a7c5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b80ee2ecec90f429, duration=2 ms] 2024-10-16 14:04:09.426 INFO 4486 --- [ocp-async-7,abc9af534212f982,2406c2b666a133dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:10.008 INFO 4486 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:04:10.008 INFO 4486 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:04:10.029 INFO 4486 --- [Check obproxy1,cf322e05b0c20df3,c5896dfe3d0583cd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:04:10.030 INFO 4486 --- [Check obproxy1,cf322e05b0c20df3,c5896dfe3d0583cd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:10.031 INFO 4486 --- [Check obproxy2,4aae931b43b143ed,01ff24e537f43828] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:04:10.031 INFO 4486 --- [Check obproxy2,4aae931b43b143ed,01ff24e537f43828] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:10.034 INFO 4486 --- [Collect obproxy2,c1e0cdd93925bd99,5fbfd3a0dc0bfc28] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:04:10.034 INFO 4486 --- [Collect obproxy2,c1e0cdd93925bd99,5fbfd3a0dc0bfc28] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:10.037 INFO 4486 --- [Collect obproxy1,a3839e6c238576cd,f06ded6262d9dfa8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:04:10.037 INFO 4486 --- [Collect obproxy1,a3839e6c238576cd,f06ded6262d9dfa8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:10.059 INFO 4486 --- [Check obproxy2,4aae931b43b143ed,01ff24e537f43828] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:04:10.059 INFO 4486 --- [Check obproxy2,4aae931b43b143ed,01ff24e537f43828] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:04:10.060 INFO 4486 --- [Collect obproxy2,c1e0cdd93925bd99,5fbfd3a0dc0bfc28] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:04:10.060 INFO 4486 --- [Check obproxy1,cf322e05b0c20df3,c5896dfe3d0583cd] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:04:10.060 INFO 4486 --- [Check obproxy1,cf322e05b0c20df3,c5896dfe3d0583cd] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:04:10.064 INFO 4486 --- [Collect obproxy1,a3839e6c238576cd,f06ded6262d9dfa8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:04:10.065 INFO 4486 --- [Collect obproxy2,c1e0cdd93925bd99,5fbfd3a0dc0bfc28] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:04:10.067 INFO 4486 --- [Collect obproxy1,a3839e6c238576cd,f06ded6262d9dfa8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:04:10.070 INFO 4486 --- [Collect obproxy2,c1e0cdd93925bd99,5fbfd3a0dc0bfc28] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:04:10.070 INFO 4486 --- [Collect obproxy1,a3839e6c238576cd,f06ded6262d9dfa8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:04:10.510 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1a4d939a75dc4e27,b48a63f19e1d2d65] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1a4d939a75dc4e27, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:10.511 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,1a4d939a75dc4e27,b48a63f19e1d2d65] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1a4d939a75dc4e27, duration=2 ms] 2024-10-16 14:04:10.575 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:10.978 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:04:10.994 INFO 4486 --- [prometheus-minute-schedule-1,1d0cd30940dfad6d,4ae813d255141bbc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:11.000 INFO 4486 --- [prometheus-minute-schedule-1,1d0cd30940dfad6d,4ae813d255141bbc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:04:11.377 INFO 4486 --- [ocp-schedules-20,a823858dd5751842,b7c1604f716d65b7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:04:11.426 INFO 4486 --- [ocp-async-1,bae822783632efbd,2670ba173e98ea73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:12.043 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,de7dd7866acec64d,a2411f3d98f4d720] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=de7dd7866acec64d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:12.043 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,de7dd7866acec64d,a2411f3d98f4d720] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=de7dd7866acec64d, duration=1 ms] 2024-10-16 14:04:12.339 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4fc5a51bea89a672,107eff224bbe4312] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4fc5a51bea89a672, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:12.339 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,4fc5a51bea89a672,107eff224bbe4312] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4fc5a51bea89a672, duration=1 ms] 2024-10-16 14:04:13.032 INFO 4486 --- [ocp-async-3,32c62161f581bbec,609495f1361cd45c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:04:13.303 INFO 4486 --- [ocp-schedules-16,a12a9a529cd43533,7762ac74c2c94218] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:04:13.303 INFO 4486 --- [ocp-schedules-16,3554fcb6bb8d7766,797a432d796f8ab5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:04:13.305 INFO 4486 --- [ocp-schedules-16,3554fcb6bb8d7766,797a432d796f8ab5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:13.335 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.335 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:13.345 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.346 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:13.377 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.378 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.378 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:04:13.379 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.379 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:13.416 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.418 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.419 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:13.420 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:04:13.420 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.424 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:04:13.425 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:13.426 INFO 4486 --- [ocp-async-1,acb8be1cfa145df5,92238c64771ded0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:13.434 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.434 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:13.446 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.455 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.456 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.457 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:13.458 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:04:13.463 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:04:13.467 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:13.469 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:13.493 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:04:13.500 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.502 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.503 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:04:13.519 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:04:13.519 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:04:13.555 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.557 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.558 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:13.562 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.568 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:04:13.568 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:04:13.571 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:13.576 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:13.577 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.578 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.578 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:04:13.587 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:04:13.587 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:04:13.598 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.598 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:13.626 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.626 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:13.654 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.654 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:13.684 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.684 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:13.712 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.712 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:13.740 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:13.767 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.768 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.769 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:13.772 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:04:13.777 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:04:13.782 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:04:13.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:04:13.809 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:04:13.958 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:04:13.993 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:13.994 INFO 4486 --- [prometheus-minute-schedule-1,9097d2bd8d68f4eb,82ee3e739c2907c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:13.995 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:13.996 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:04:14.002 INFO 4486 --- [prometheus-minute-schedule-1,9097d2bd8d68f4eb,82ee3e739c2907c5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:14.034 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:04:14.034 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:04:14.651 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058580, nodeCount=1000, affectRows=1000 2024-10-16 14:04:14.656 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058580, nodeCount=999, validCount=999 2024-10-16 14:04:14.661 ERROR 4486 --- [metric-archive-second-13,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=706681) Duplicate entry '669741696490737664-1729058580' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:04:14.660692] [YB420AC531B8-00060B7DC0C8A164-0-0]; nested exception is java.sql.BatchUpdateException: (conn=706681) Duplicate entry '669741696490737664-1729058580' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:04:14.660692] [YB420AC531B8-00060B7DC0C8A164-0-0] 2024-10-16 14:04:14.683 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058580, nodeCount=1000, affectRows=1000 2024-10-16 14:04:14.686 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058580, nodeCount=1000, validCount=1000 2024-10-16 14:04:14.688 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058580, nodeCount=1000, affectRows=1000 2024-10-16 14:04:14.691 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058580, nodeCount=1000, validCount=1000 2024-10-16 14:04:14.870 INFO 4486 --- [ocp-schedules-4,cb7a80cc3cdb30dc,c2b453035b721d69] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:04:14.908 WARN 4486 --- [scanner-13327017-1,cb7a80cc3cdb30dc,cebee88096d8cbcd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:04:14.958 INFO 4486 --- [alarm-detect-13,cb7a80cc3cdb30dc,e9624d69c6dd1135] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058625 ~ 1729058644 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058640, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058640, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058640=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:04:14.972 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058580, nodeCount=449, affectRows=449 2024-10-16 14:04:14.975 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058580, nodeCount=447, validCount=447 2024-10-16 14:04:15.292 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058580, nodeCount=250, affectRows=250 2024-10-16 14:04:15.326 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058580, nodeCount=227, affectRows=227 2024-10-16 14:04:15.427 INFO 4486 --- [ocp-async-2,776d5611188a845f,b42435820488a3b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:15.680 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058580, nodeCount=227, validCount=227 2024-10-16 14:04:15.693 INFO 4486 --- [metric-archive-async-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058580, nodeCount=250, validCount=250 2024-10-16 14:04:16.577 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:16.994 INFO 4486 --- [prometheus-minute-schedule-1,da9290f306dfb0ff,d12163bdf95d1afd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:17.000 INFO 4486 --- [prometheus-minute-schedule-1,da9290f306dfb0ff,d12163bdf95d1afd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:17.069 INFO 4486 --- [ocp-schedules-22,667bd633ade2ef0f,55edb4c30efb8179] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:04:17.196 INFO 4486 --- [ocp-schedules-13,e9761a3c515565c9,82f59d30d67585ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:04:17.297 INFO 4486 --- [ocp-schedules-26,6da28fc71fc602ac,d169eccc6e08f101] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:04:17.426 INFO 4486 --- [ocp-async-6,9a96af5d14525fe9,3709ec3f2c84a574] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:18.483 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1a4d802e0d5a7c1c,ab350cf35b9a61f8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1a4d802e0d5a7c1c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:18.483 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,1a4d802e0d5a7c1c,ab350cf35b9a61f8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1a4d802e0d5a7c1c, duration=2 ms] 2024-10-16 14:04:19.303 INFO 4486 --- [ocp-schedules-1,fd39b38aa017e715,fc9a0e328451116a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:04:19.427 INFO 4486 --- [ocp-async-9,73eb0b4cd893782b,c872d453d5621d99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:19.578 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:19.994 INFO 4486 --- [prometheus-minute-schedule-1,0ace30d8f37987e9,58b907974e2a720f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:20.001 INFO 4486 --- [prometheus-minute-schedule-1,0ace30d8f37987e9,58b907974e2a720f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:20.516 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0fe27543e744c15f,c07683be2a6ac680] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0fe27543e744c15f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:20.516 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,0fe27543e744c15f,c07683be2a6ac680] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0fe27543e744c15f, duration=2 ms] 2024-10-16 14:04:20.814 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 14:04:20.996 INFO 4486 --- [prometheus-cache-monitor1,10606551bcc4d1c4,cc5d2ee7461f0332] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13796,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8129} 2024-10-16 14:04:20.996 INFO 4486 --- [prometheus-cache-monitor1,10606551bcc4d1c4,cc5d2ee7461f0332] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:04:21.384 INFO 4486 --- [ocp-schedules-24,c5e80c09d1be7275,06b3ecf6ec48d6ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:04:21.427 INFO 4486 --- [ocp-async-5,6bd3b7e35127a68b,182ecb7eba153885] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:22.049 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6f784dbb98f383fd,3645b7ad592ce75c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6f784dbb98f383fd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:22.050 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6f784dbb98f383fd,3645b7ad592ce75c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6f784dbb98f383fd, duration=2 ms] 2024-10-16 14:04:22.345 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,464cabc6cd807f97,51f4380cce07ab2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=464cabc6cd807f97, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:22.345 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,464cabc6cd807f97,51f4380cce07ab2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=464cabc6cd807f97, duration=1 ms] 2024-10-16 14:04:22.579 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:22.994 INFO 4486 --- [prometheus-minute-schedule-1,fe750b59214e55a5,691d6f2884d5de09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:23.000 INFO 4486 --- [prometheus-minute-schedule-1,fe750b59214e55a5,691d6f2884d5de09] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:23.032 INFO 4486 --- [ocp-async-1,123a9671bb52dd91,5df4804e105deed1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:04:23.427 INFO 4486 --- [ocp-async-8,724fcabd4ffb6df4,d39b4efa6be9413e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:23.900 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ec5c10da52cf7335,6cf02a5cbafe43f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=ec5c10da52cf7335, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:04:23.900 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ec5c10da52cf7335,6cf02a5cbafe43f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=ec5c10da52cf7335, duration=1 ms] 2024-10-16 14:04:24.092 INFO 4486 --- [ocp-schedules-18,a3a3a65a198da027,4e85ddbcfa6c7c48] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:04:25.023 INFO 4486 --- [ocp-schedules-21,d49d77c9c79a87b7,24bcf835e91767a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:04:25.057 WARN 4486 --- [scanner-13327217-1,d49d77c9c79a87b7,ded1ef56ff7d6dfc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:04:25.137 INFO 4486 --- [alarm-detect-2,d49d77c9c79a87b7,82c18c11ad5a969f] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058640 ~ 1729058655 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058655, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058655, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058655=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:04:25.321 INFO 4486 --- [ocp-schedules-1,f3dd456e4046486e,5db9a343a56fd483] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:04:25.427 INFO 4486 --- [ocp-async-9,7fb3f65fe37b1e3c,cd603f8706d7e3e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:25.580 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:25.994 INFO 4486 --- [prometheus-minute-schedule-1,0bcde4f43ef3a9fb,58c8157e976912a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:25.999 INFO 4486 --- [prometheus-minute-schedule-1,0bcde4f43ef3a9fb,58c8157e976912a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:26.999 INFO 4486 --- [ocp-schedules-8,3deab8e144a9135b,86686d6fbda7fdfe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:04:26.999 INFO 4486 --- [ocp-schedules-8,3deab8e144a9135b,86686d6fbda7fdfe] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:04:27.034 INFO 4486 --- [Collect all tenants major compactions1,3deab8e144a9135b,57e6f7ec82d53938] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:04:27.034 INFO 4486 --- [Collect all tenants major compactions3,3deab8e144a9135b,45eb6cbf1aab380a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:04:27.034 INFO 4486 --- [Collect all tenants major compactions5,3deab8e144a9135b,6ff60dcd57472dae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:04:27.034 INFO 4486 --- [Collect all tenants major compactions2,3deab8e144a9135b,b5dd20e39625ea28] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:04:27.035 INFO 4486 --- [Collect all tenants major compactions4,3deab8e144a9135b,a204f4c32866ca0b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:04:27.047 INFO 4486 --- [Collect all tenants major compactions2,3deab8e144a9135b,b5dd20e39625ea28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:27.047 INFO 4486 --- [Collect all tenants major compactions1,3deab8e144a9135b,57e6f7ec82d53938] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:27.049 INFO 4486 --- [Collect all tenants major compactions1,3deab8e144a9135b,57e6f7ec82d53938] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:04:27.049 INFO 4486 --- [Collect all tenants major compactions2,3deab8e144a9135b,b5dd20e39625ea28] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:04:27.053 INFO 4486 --- [Collect all tenants major compactions2,3deab8e144a9135b,b5dd20e39625ea28] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:04:27.053 INFO 4486 --- [Collect all tenants major compactions1,3deab8e144a9135b,57e6f7ec82d53938] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:04:27.053 INFO 4486 --- [Collect all tenants major compactions1,3deab8e144a9135b,57e6f7ec82d53938] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:04:27.053 INFO 4486 --- [Collect all tenants major compactions2,3deab8e144a9135b,b5dd20e39625ea28] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:04:27.054 INFO 4486 --- [Collect all tenants major compactions3,3deab8e144a9135b,45eb6cbf1aab380a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:27.056 INFO 4486 --- [Collect all tenants major compactions3,3deab8e144a9135b,45eb6cbf1aab380a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:04:27.060 INFO 4486 --- [Collect all tenants major compactions3,3deab8e144a9135b,45eb6cbf1aab380a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:04:27.060 INFO 4486 --- [Collect all tenants major compactions3,3deab8e144a9135b,45eb6cbf1aab380a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:04:27.064 INFO 4486 --- [Collect all tenants major compactions4,3deab8e144a9135b,a204f4c32866ca0b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:27.065 INFO 4486 --- [Collect all tenants major compactions5,3deab8e144a9135b,6ff60dcd57472dae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:27.066 INFO 4486 --- [Collect all tenants major compactions4,3deab8e144a9135b,a204f4c32866ca0b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:04:27.067 INFO 4486 --- [Collect all tenants major compactions5,3deab8e144a9135b,6ff60dcd57472dae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:04:27.069 INFO 4486 --- [Collect all tenants major compactions4,3deab8e144a9135b,a204f4c32866ca0b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:04:27.070 INFO 4486 --- [Collect all tenants major compactions4,3deab8e144a9135b,a204f4c32866ca0b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:04:27.071 INFO 4486 --- [Collect all tenants major compactions5,3deab8e144a9135b,6ff60dcd57472dae] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:04:27.071 INFO 4486 --- [Collect all tenants major compactions5,3deab8e144a9135b,6ff60dcd57472dae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:04:27.072 INFO 4486 --- [ocp-schedules-8,3deab8e144a9135b,86686d6fbda7fdfe] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:04:27.428 INFO 4486 --- [ocp-async-10,8a7b438ca7ff7efa,d95aaf9437734cae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:27.970 INFO 4486 --- [ocp-schedules-28,c0a8bbff01a2343d,726000c2544bfcce] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:04:27.970+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:04:27.972 INFO 4486 --- [ocp-schedules-28,c0a8bbff01a2343d,726000c2544bfcce] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:04:27.973 INFO 4486 --- [ocp-schedules-28,c0a8bbff01a2343d,726000c2544bfcce] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:04:27.973 INFO 4486 --- [ocp-schedules-28,c0a8bbff01a2343d,726000c2544bfcce] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:04:28.489 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,615503215b7e3f79,1863a963aa3f2384] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=615503215b7e3f79, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:28.489 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,615503215b7e3f79,1863a963aa3f2384] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=615503215b7e3f79, duration=2 ms] 2024-10-16 14:04:28.581 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:28.994 INFO 4486 --- [prometheus-minute-schedule-1,0c233da55f1b3bf7,d3b792f1fbc0ece1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:29.000 INFO 4486 --- [prometheus-minute-schedule-1,0c233da55f1b3bf7,d3b792f1fbc0ece1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:29.428 INFO 4486 --- [ocp-async-6,1a27a2943a733037,1edc54247dd31c84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:29.544 INFO 4486 --- [ocp-schedules-3,a1c9115ed4331877,43de38ebe8464e9a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 14:04:29.547 INFO 4486 --- [ocp-schedules-3,a1c9115ed4331877,43de38ebe8464e9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:29.585 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:04:29.585 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.590 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:04:29.599 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.599 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:04:29.605 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:04:29.605 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.607 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.608 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:04:29.611 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:04:29.615 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.616 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:04:29.624 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.625 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:04:29.626 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.627 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:04:29.636 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.637 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:29.641 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.641 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:04:29.642 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.643 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:04:29.645 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.646 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:29.646 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:04:29.655 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.656 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.656 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:04:29.656 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:29.660 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.661 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:04:29.668 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.668 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.669 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:04:29.669 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:29.677 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.678 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:04:29.679 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.680 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:04:29.686 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.687 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.688 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.689 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.689 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.690 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.690 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.691 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.691 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:04:29.691 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.692 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.692 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.693 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.693 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:04:29.693 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.694 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.695 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.696 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.696 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.697 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.697 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.697 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.698 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:04:29.698 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.699 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.700 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.701 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.703 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.703 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.704 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.705 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 14:04:29.709 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:04:29.710 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.711 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:04:29.715 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.717 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.719 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.727 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:29.727 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.728 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:04:29.729 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.729 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.730 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:04:29.739 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.739 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.739 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.739 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 14:04:29.740 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.740 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 14:04:29.740 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.740 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 14:04:29.740 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.740 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 14:04:29.740 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:29.743 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.744 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 14:04:29.744 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:04:29.745 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.746 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.749 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.750 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.751 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.752 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:29.753 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.755 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.758 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.759 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.762 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.763 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.763 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.764 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:29.766 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.767 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.768 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 14:04:29.771 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:04:29.797 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.798 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.801 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:29.801 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.802 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.802 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.803 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:04:29.805 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.806 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.809 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.810 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.813 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.813 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.813 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 14:04:29.813 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.813 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 14:04:29.813 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.813 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 14:04:29.815 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.817 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 14:04:29.817 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 14:04:29.818 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.819 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.822 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.824 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:04:29.827 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.828 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:04:29.831 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.832 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:04:29.834 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 14:04:29.837 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:04:29.892 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:29.893 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:29.894 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:04:29.906 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.906 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 14:04:29.906 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.906 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 14:04:29.906 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:04:29.906 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 14:04:29.909 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 14:04:29.909 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:04:30.103 INFO 4486 --- [prometheus-second-schedule-1,780c7a18a67fd35f,a23683175f406a0a] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:04:30.363 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9b3703ea93ad3805,30ce9d86c788f627] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=9b3703ea93ad3805, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:04:30.363 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9b3703ea93ad3805,30ce9d86c788f627] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=9b3703ea93ad3805, duration=1 ms] 2024-10-16 14:04:30.520 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e33ee5a2966b09de,5f357bb71f083ba2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e33ee5a2966b09de, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:30.520 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,e33ee5a2966b09de,5f357bb71f083ba2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e33ee5a2966b09de, duration=1 ms] 2024-10-16 14:04:31.390 INFO 4486 --- [ocp-schedules-17,fb8030bc13ebb2fb,8c9ae78c232226f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:04:31.428 INFO 4486 --- [ocp-async-4,11655ee4de384be7,79b871ebf89e002b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:31.582 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:31.994 INFO 4486 --- [prometheus-minute-schedule-1,82141c67fccbef0f,1c44cde83affc31d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:31.999 INFO 4486 --- [prometheus-minute-schedule-1,82141c67fccbef0f,1c44cde83affc31d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:32.056 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,06b7ddefd6bb0a71,c98c1670a0b0fed7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=06b7ddefd6bb0a71, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:32.056 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,06b7ddefd6bb0a71,c98c1670a0b0fed7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=06b7ddefd6bb0a71, duration=1 ms] 2024-10-16 14:04:32.349 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,af59e738db9f8a4a,bb09347d64e6954b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=af59e738db9f8a4a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:32.349 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,af59e738db9f8a4a,bb09347d64e6954b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=af59e738db9f8a4a, duration=1 ms] 2024-10-16 14:04:33.032 INFO 4486 --- [ocp-async-8,3b66806f7c9222a1,26ab2e5a34de14dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:04:33.428 INFO 4486 --- [ocp-async-10,a46532bd403b1cb1,e621ae5fb2f9f38a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:34.583 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:34.994 INFO 4486 --- [prometheus-minute-schedule-1,f9ecc913c398b05c,bb770fbdea2ce7d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:35.000 INFO 4486 --- [prometheus-minute-schedule-1,f9ecc913c398b05c,bb770fbdea2ce7d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:35.247 INFO 4486 --- [ocp-schedules-30,4e52490f5e708be7,02568684d4237b4f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:04:35.288 WARN 4486 --- [scanner-13327569-1,4e52490f5e708be7,6d9f5b369630f64e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:04:35.428 INFO 4486 --- [ocp-async-3,a923f8e630fc3290,10a12ff0b186c702] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:37.063 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:04:37.064 INFO 4486 --- [ocp-schedules-8,85ff173ee5ba037b,b879bd463a599de2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:04:37.066 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:04:37.066 INFO 4486 --- [ocp-schedules-6,c5c29f9b917ba10c,33ea4db4ce813d90] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:04:37.069 INFO 4486 --- [ocp-schedules-6,c5c29f9b917ba10c,33ea4db4ce813d90] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:37.069 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:04:37.071 INFO 4486 --- [ocp-schedules-8,85ff173ee5ba037b,b879bd463a599de2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:04:37.081 INFO 4486 --- [ocp-schedules-4,b49500ab963e561f,2b2aa7d52fd42148] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:04:37.083 INFO 4486 --- [ocp-schedules-4,b49500ab963e561f,2b2aa7d52fd42148] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:04:37.091 INFO 4486 --- [ocp-schedules-23,ab70a7dc341adc75,df5b5a15fff6a147] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:04:37.091 INFO 4486 --- [ocp-schedules-26,00e2760faad2ffaf,c9989c41402d614c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:04:37.092 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:04:37.093 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:04:37.093 INFO 4486 --- [host-check-agent-status2,ab70a7dc341adc75,ee4115d8e55dd706] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:04:37.093 INFO 4486 --- [host-check-agent-status1,ab70a7dc341adc75,2510b758a6284fb7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:04:37.093 INFO 4486 --- [ocp-schedules-13,39f88b8e8d025cab,a99a977f3329c6d4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:04:37.094 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:04:37.094 INFO 4486 --- [host-check-agent-status3,ab70a7dc341adc75,be1ec5d859c08d77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:04:37.096 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:04:37.096 INFO 4486 --- [host-check-agent-status4,ab70a7dc341adc75,df4ca66dd8599f83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:04:37.098 INFO 4486 --- [host-check-clockdiff2,39f88b8e8d025cab,33434a96700af58a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:04:37.099 INFO 4486 --- [host-check-clockdiff3,39f88b8e8d025cab,9204723dea8e8f0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:04:37.099 INFO 4486 --- [host-check-clockdiff4,39f88b8e8d025cab,074ee45b9d8ef223] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:04:37.099 INFO 4486 --- [host-check-clockdiff1,39f88b8e8d025cab,47cc6173a065e708] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:04:37.103 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:37.104 INFO 4486 --- [host-check-agent-status4,ab70a7dc341adc75,df4ca66dd8599f83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:37.105 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:37.105 INFO 4486 --- [host-check-agent-status2,ab70a7dc341adc75,ee4115d8e55dd706] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:37.105 INFO 4486 --- [host-check-agent-status1,ab70a7dc341adc75,2510b758a6284fb7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:37.105 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:37.106 INFO 4486 --- [host-check-agent-status3,ab70a7dc341adc75,be1ec5d859c08d77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:37.106 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:37.107 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:37.107 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:37.107 INFO 4486 --- [host-check-agent-status1,ab70a7dc341adc75,2510b758a6284fb7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:37.107 INFO 4486 --- [host-check-agent-status4,ab70a7dc341adc75,df4ca66dd8599f83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:37.107 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:37.108 INFO 4486 --- [host-check-agent-status2,ab70a7dc341adc75,ee4115d8e55dd706] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:37.108 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:37.108 INFO 4486 --- [host-check-agent-status3,ab70a7dc341adc75,be1ec5d859c08d77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:37.141 INFO 4486 --- [host-check-clockdiff1,39f88b8e8d025cab,47cc6173a065e708] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058677 0 0 , error= 2024-10-16 14:04:37.143 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:37.146 INFO 4486 --- [host-check-clockdiff3,39f88b8e8d025cab,9204723dea8e8f0a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058677 4 4 , error= 2024-10-16 14:04:37.155 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.156 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:37.157 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:04:37.165 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:37.167 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.167 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.168 INFO 4486 --- [host-check-agent-status4,ab70a7dc341adc75,df4ca66dd8599f83] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:04:37.168 INFO 4486 --- [host-check-agent-status2,ab70a7dc341adc75,ee4115d8e55dd706] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:04:37.168 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:04:37.168 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:04:37.169 INFO 4486 --- [host-check-agent-status3,ab70a7dc341adc75,be1ec5d859c08d77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:04:37.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:37.169 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:04:37.177 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:04:37.177 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:04:37.178 INFO 4486 --- [host-check-agent-status1,ab70a7dc341adc75,2510b758a6284fb7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:04:37.179 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:04:37.182253882+08:00 2024-10-16 14:04:37.179 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:04:37.182129913+08:00 2024-10-16 14:04:37.179 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:04:37.182210916+08:00 2024-10-16 14:04:37.179 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:04:37.179+08:00 2024-10-16 14:04:37.179 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:04:37.179+08:00 2024-10-16 14:04:37.179 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:04:37.179+08:00 2024-10-16 14:04:37.179 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:04:37.179269896+08:00 2024-10-16 14:04:37.180 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:04:37.180 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:04:37.179+08:00 2024-10-16 14:04:37.181 INFO 4486 --- [host-check-agent-status3,ab70a7dc341adc75,be1ec5d859c08d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:04:37.181 INFO 4486 --- [host-check-agent-status3,ab70a7dc341adc75,be1ec5d859c08d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:04:37.181 INFO 4486 --- [host-check-agent-status3,ab70a7dc341adc75,be1ec5d859c08d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:04:37.179+08:00 and status AVAILABLE 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:04:37.179+08:00 and status AVAILABLE 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-agent-status4,ab70a7dc341adc75,df4ca66dd8599f83] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-agent-status4,ab70a7dc341adc75,df4ca66dd8599f83] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-agent-status4,ab70a7dc341adc75,df4ca66dd8599f83] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:04:37.179+08:00 and status AVAILABLE 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:04:37.179+08:00 and status AVAILABLE 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-agent-status2,ab70a7dc341adc75,ee4115d8e55dd706] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-agent-status2,ab70a7dc341adc75,ee4115d8e55dd706] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:04:37.182 INFO 4486 --- [host-check-agent-status2,ab70a7dc341adc75,ee4115d8e55dd706] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:04:37.184 INFO 4486 --- [host-check-agent-status1,ab70a7dc341adc75,2510b758a6284fb7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:04:37.184 INFO 4486 --- [host-check-agent-status1,ab70a7dc341adc75,2510b758a6284fb7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:04:37.184 INFO 4486 --- [host-check-agent-status1,ab70a7dc341adc75,2510b758a6284fb7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:04:37.184 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:04:37.185 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:04:37.186 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:04:37.186 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:04:37.187 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.188 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.188 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:04:37.189 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:37.189 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.190 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.192 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:04:37.194 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:37.194 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:37.194 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:37.195 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:37.195 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:37.196 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:37.196 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:37.196 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:37.201 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.202 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.202 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.202 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.202 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.202 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.202 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:37.203 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.218 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:37.219 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:37.220 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:04:37.220 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:37.220 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:04:37.229 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.230 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:04:37.232 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.235 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:04:37.249 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.250 INFO 4486 --- [ocp-schedules-9,7a3514ce0a799051,ddf0c270e7e23c38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:04:37.252 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:04:37.253 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:04:37.264 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:04:37.265 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:04:37.266 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:04:37.274 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:04:37.275 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:04:37.283 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:37.284 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:37.285 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:04:37.290 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:37.302 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:04:37.314 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:04:37.317 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:04:37.325 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:04:37.370 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:04:37.370 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:04:37.370 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:04:37.382 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:04:37.382 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:04:37.385 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:37.389 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:37.397 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:37.400 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:04:37.414 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:04:37.424 INFO 4486 --- [host-check-host-status3,00e2760faad2ffaf,bf3905b3f86258e5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:04:37.429 INFO 4486 --- [ocp-async-9,797e9bb5bcf7132f,663902df0906062e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:37.429 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:04:37.438 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:04:37.441 INFO 4486 --- [host-check-host-status4,00e2760faad2ffaf,f71456ecfdee6f98] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:04:37.450 INFO 4486 --- [host-check-host-status2,00e2760faad2ffaf,0c191ec483fd59c3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:04:37.453 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:04:37.464 INFO 4486 --- [host-check-host-status1,00e2760faad2ffaf,bac1fedb58818097] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:04:37.480 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:37.482 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.483 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:04:37.484 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.484 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.486 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:04:37.496 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.496 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.496 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.496 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.496 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.497 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.523 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:04:37.565 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:04:37.584 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:37.613 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:37.630 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:37.632 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.634 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:04:37.635 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:04:37.636 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.638 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:04:37.639 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.639 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.640 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:04:37.649 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.649 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.649 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.649 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.649 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.651 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.658 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:04:37.870 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:04:37.919 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:37.922 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.923 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:04:37.925 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.925 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.927 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:04:37.943 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.943 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.943 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.943 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.943 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:37.943 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:37.962 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:04:37.964 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:04:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:04:37.994 INFO 4486 --- [prometheus-minute-schedule-1,400bdbdfeabea554,054a65d2467c5a12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:38.001 INFO 4486 --- [prometheus-minute-schedule-1,400bdbdfeabea554,054a65d2467c5a12] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:38.062 INFO 4486 --- [host-check-clockdiff2,39f88b8e8d025cab,33434a96700af58a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058678 4 4 , error= 2024-10-16 14:04:38.169 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:04:38.212 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:38.214 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:38.215 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:04:38.216 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:38.217 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:38.218 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:04:38.229 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:38.229 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:38.229 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:38.229 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:38.229 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:04:38.229 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:38.294 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:04:38.489 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:04:38.298061, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:04:38.490 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:04:38.298061, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:04:38.509 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,5928b70aa06016b0,200debc60e8d8d41] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5928b70aa06016b0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:38.510 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,5928b70aa06016b0,200debc60e8d8d41] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5928b70aa06016b0, duration=3 ms] 2024-10-16 14:04:38.531 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:04:38.298061} } 2024-10-16 14:04:38.585 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:04:38.593 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:04:38.596 INFO 4486 --- [ocp-schedules-27,e93c4c2989b64592,56d0e389bb73fe22] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:04:38.920 INFO 4486 --- [host-check-clockdiff4,39f88b8e8d025cab,074ee45b9d8ef223] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058678 4 4 , error= 2024-10-16 14:04:39.307 INFO 4486 --- [ocp-schedules-28,3992c8fae3315919,b5c77ee505f15cd5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:04:39.429 INFO 4486 --- [ocp-async-5,0347c32463eb7082,c07b00fe222210a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:39.475 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,928a714d191d9437,a6479b318d217d7a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=928a714d191d9437, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:04:39.476 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,928a714d191d9437,a6479b318d217d7a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=928a714d191d9437, duration=2 ms] 2024-10-16 14:04:40.006 INFO 4486 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:04:40.006 INFO 4486 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:04:40.025 INFO 4486 --- [Check obproxy1,6601a11b912272dd,529b16e101560170] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:04:40.025 INFO 4486 --- [Check obproxy2,739e53be1b7da91b,53402076a49bedeb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:04:40.025 INFO 4486 --- [Check obproxy2,739e53be1b7da91b,53402076a49bedeb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:40.025 INFO 4486 --- [Check obproxy1,6601a11b912272dd,529b16e101560170] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:40.029 INFO 4486 --- [Collect obproxy1,1cb36c43cdc3f510,d8dcdbc394027e72] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:04:40.029 INFO 4486 --- [Collect obproxy1,1cb36c43cdc3f510,d8dcdbc394027e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:40.030 INFO 4486 --- [Collect obproxy2,b79a142f1499d5ce,d0f9e98abd374d74] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:04:40.030 INFO 4486 --- [Collect obproxy2,b79a142f1499d5ce,d0f9e98abd374d74] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:04:40.048 INFO 4486 --- [Check obproxy2,739e53be1b7da91b,53402076a49bedeb] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:04:40.049 INFO 4486 --- [Check obproxy2,739e53be1b7da91b,53402076a49bedeb] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:04:40.050 INFO 4486 --- [Collect obproxy2,b79a142f1499d5ce,d0f9e98abd374d74] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:04:40.052 INFO 4486 --- [Check obproxy1,6601a11b912272dd,529b16e101560170] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:04:40.052 INFO 4486 --- [Check obproxy1,6601a11b912272dd,529b16e101560170] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:04:40.055 INFO 4486 --- [Collect obproxy2,b79a142f1499d5ce,d0f9e98abd374d74] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:04:40.060 INFO 4486 --- [Collect obproxy2,b79a142f1499d5ce,d0f9e98abd374d74] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:04:40.062 INFO 4486 --- [Collect obproxy1,1cb36c43cdc3f510,d8dcdbc394027e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:04:40.065 INFO 4486 --- [Collect obproxy1,1cb36c43cdc3f510,d8dcdbc394027e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:04:40.070 INFO 4486 --- [Collect obproxy1,1cb36c43cdc3f510,d8dcdbc394027e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:04:40.527 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,206c96362a7f89f2,905cd674f21e8cd7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=206c96362a7f89f2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:40.527 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,206c96362a7f89f2,905cd674f21e8cd7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=206c96362a7f89f2, duration=1 ms] 2024-10-16 14:04:40.585 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:40.994 INFO 4486 --- [prometheus-minute-schedule-1,3ed8159947cb1c22,c75c61ce40729573] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:40.995 INFO 4486 --- [prometheus-hour-schedule-1,e957b2b492b8a8cc,010c81a23c65cdb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:04:40.999 INFO 4486 --- [prometheus-minute-schedule-1,3ed8159947cb1c22,c75c61ce40729573] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:41.395 INFO 4486 --- [ocp-schedules-24,aad444d18521d436,080b38670d1df782] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:04:41.429 INFO 4486 --- [ocp-async-3,0cd92facf65dae19,555a435bf6831b97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:42.061 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8699b1117bddd324,4ba664269ef22d33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8699b1117bddd324, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:42.062 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8699b1117bddd324,4ba664269ef22d33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8699b1117bddd324, duration=1 ms] 2024-10-16 14:04:42.196 INFO 4486 --- [ocp-schedules-23,07ae22fd007433b6,39288aa6ccb414aa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:04:42.354 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,71c3468678b7c4b5,f5f342f09aa14ea6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=71c3468678b7c4b5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:42.354 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,71c3468678b7c4b5,f5f342f09aa14ea6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=71c3468678b7c4b5, duration=1 ms] 2024-10-16 14:04:43.032 INFO 4486 --- [ocp-async-2,4eb0e7cfaf6030ec,17058111c42b2670] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:04:43.429 INFO 4486 --- [ocp-async-4,23f337ff764582e9,8c50c27ea13a0a4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:43.560 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 14:04:43.561 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 14:04:43.562 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.563 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:04:43.564 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.564 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.565 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.566 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 14:04:43.582 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.583 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.584 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.584 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 14:04:43.585 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:43.588 INFO 4486 --- [ocp-schedules-13,6cd9c429c9e8513b,41722bbbd9719d99] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:04:43.589 INFO 4486 --- [ocp-schedules-13,6cd9c429c9e8513b,41722bbbd9719d99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:43.590 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:04:43.590 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:04:43.590 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:04:43.597 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.598 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.599 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.600 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 14:04:43.617 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.617 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.618 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.619 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 14:04:43.634 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.635 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.636 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.637 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 14:04:43.645 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.646 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.647 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.647 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 14:04:43.654 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.654 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.655 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.657 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 14:04:43.663 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.664 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.664 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.665 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 14:04:43.673 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.673 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.674 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.675 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 14:04:43.682 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.682 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.684 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.684 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 14:04:43.692 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.693 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.694 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.695 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 14:04:43.704 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.705 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.706 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.707 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 14:04:43.721 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.722 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.723 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.723 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 14:04:43.730 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.731 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.731 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.732 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 14:04:43.738 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.739 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.739 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.740 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 14:04:43.747 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.748 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.748 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.749 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 14:04:43.754 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.755 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.756 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.756 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 14:04:43.761 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.762 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.763 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.763 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 14:04:43.767 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.768 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.769 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.769 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 14:04:43.774 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.774 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.775 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.775 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 14:04:43.780 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.780 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.781 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.782 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 14:04:43.787 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.788 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.789 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.789 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 14:04:43.794 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.795 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.795 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.796 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 14:04:43.800 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.801 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.802 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.802 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 14:04:43.807 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.807 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.808 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.808 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 14:04:43.813 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.813 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.814 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.814 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 14:04:43.820 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.820 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.821 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.821 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 14:04:43.826 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.827 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.828 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.828 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 14:04:43.833 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.834 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.834 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.835 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 14:04:43.841 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.841 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.842 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.842 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 14:04:43.848 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.848 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.849 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.849 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 14:04:43.854 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.854 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.855 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.855 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 14:04:43.863 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.863 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.864 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.865 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 14:04:43.869 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.869 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.870 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.870 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 14:04:43.879 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.879 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.880 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.880 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 14:04:43.889 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.890 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.891 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.891 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 14:04:43.897 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.897 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.898 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.898 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 14:04:43.904 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.904 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.905 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.905 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 14:04:43.910 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.910 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.911 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.912 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 14:04:43.916 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.916 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.917 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.917 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 14:04:43.922 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.923 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.923 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.924 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 14:04:43.929 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.930 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.930 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.931 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 14:04:43.941 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.942 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.943 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.943 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 14:04:43.966 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.967 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.967 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.968 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 14:04:43.978 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.979 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.980 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.981 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 14:04:43.988 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.989 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:43.989 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.990 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 14:04:43.994 INFO 4486 --- [prometheus-minute-schedule-1,cd3e67eac3d7b172,d11b6fc7d44a9ad6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:43.999 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:43.999 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:44.000 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.001 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 14:04:44.002 INFO 4486 --- [prometheus-minute-schedule-1,cd3e67eac3d7b172,d11b6fc7d44a9ad6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:44.006 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.006 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:44.007 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.008 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 14:04:44.014 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.014 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:44.016 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.017 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 14:04:44.023 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 14:04:44.023 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 14:04:44.025 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.026 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:04:44.027 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.028 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:44.028 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.029 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 14:04:44.035 INFO 4486 --- [ocp-schedules-16,3554fcb6bb8d7766,797a432d796f8ab5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:04:44.035 INFO 4486 --- [ocp-schedules-16,e44a539ff847f11f,4bb0fab6a8e58759] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:04:44.035 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.036 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:04:44.036 INFO 4486 --- [ocp-schedules-16,e44a539ff847f11f,4bb0fab6a8e58759] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:04:44.037 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.037 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 14:04:44.042 INFO 4486 --- [ocp-schedules-24,2fcfba2f5eb4517a,daaaeee60224b4ad] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 14:04:44.064 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.064 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:44.074 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.075 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:44.101 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.102 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.103 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:04:44.104 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.105 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:44.132 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.132 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.133 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:44.134 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:04:44.134 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.138 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:04:44.140 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:44.149 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.149 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:44.160 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.171 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.172 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.172 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:44.173 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:04:44.179 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:04:44.185 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:44.187 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:44.210 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:04:44.215 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.216 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.217 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:04:44.242 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:04:44.242 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:04:44.265 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.267 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.268 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:04:44.271 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.274 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:04:44.277 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:44.288 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:04:44.298 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.298 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.299 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:04:44.307 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.307 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:44.309 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:04:44.309 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:04:44.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:44.361 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.361 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:44.390 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.390 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:04:44.416 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.416 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:04:44.443 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:04:44.466 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.468 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.469 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:04:44.473 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:04:44.477 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:04:44.481 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:04:44.483 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:04:44.508 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:04:44.641 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:04:44.668 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:04:44.670 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:04:44.671 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:04:44.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:04:44.704 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:04:45.419 INFO 4486 --- [ocp-schedules-7,f9d00e00a1c98c2f,1ec781da381d40c4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:04:45.430 INFO 4486 --- [ocp-async-10,652dce4a948102d0,c1288e8f695035f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:45.476 WARN 4486 --- [scanner-13327816-1,f9d00e00a1c98c2f,540229b4790388e6] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:04:45.532 INFO 4486 --- [alarm-detect-15,f9d00e00a1c98c2f,f5a4c006623988fd] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058655 ~ 1729058675 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058670, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058670, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058670=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:04:45.610 INFO 4486 --- [ocp-schedules-11,eab00d5e1a446b2c,5a0b9d0c17cbca3f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:04:45.620 INFO 4486 --- [ocp-schedules-11,eab00d5e1a446b2c,5a0b9d0c17cbca3f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:04:45.620 INFO 4486 --- [ocp-schedules-11,eab00d5e1a446b2c,5a0b9d0c17cbca3f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:04:45.649 INFO 4486 --- [ocp-schedules-8,e9814316c4fdf25e,cd55b929135d2d19] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:04:45.649 INFO 4486 --- [ocp-schedules-8,e9814316c4fdf25e,cd55b929135d2d19] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:04:46.586 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:46.994 INFO 4486 --- [prometheus-minute-schedule-1,f65be7c759fef02b,87139b96781c82ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:47.000 INFO 4486 --- [prometheus-minute-schedule-1,f65be7c759fef02b,87139b96781c82ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:47.070 INFO 4486 --- [ocp-schedules-6,c5c29f9b917ba10c,33ea4db4ce813d90] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:04:47.177 INFO 4486 --- [ocp-schedules-4,b49500ab963e561f,2b2aa7d52fd42148] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:04:47.301 INFO 4486 --- [ocp-schedules-28,ebeee86ddedf99bc,e9a682c94dc8f740] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:04:47.430 INFO 4486 --- [ocp-async-3,288a251a1f6001e0,5d25444422808d15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:48.516 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,bf2559d202546e16,a68e3ec24733699e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bf2559d202546e16, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:48.516 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,bf2559d202546e16,a68e3ec24733699e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bf2559d202546e16, duration=2 ms] 2024-10-16 14:04:49.430 INFO 4486 --- [ocp-async-4,25fbb760c4b2e305,0b9386bc6dfcc093] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:49.587 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:49.994 INFO 4486 --- [prometheus-minute-schedule-1,0b7fb03a25cf148b,8fcccd298379f58a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:49.999 INFO 4486 --- [prometheus-minute-schedule-1,0b7fb03a25cf148b,8fcccd298379f58a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:50.533 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,c23af1d3c878acb9,e9e0d745900089a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c23af1d3c878acb9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:50.534 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,c23af1d3c878acb9,e9e0d745900089a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c23af1d3c878acb9, duration=1 ms] 2024-10-16 14:04:50.586 INFO 4486 --- [ocp-schedules-29,10e2a5edef5ce5ae,7d4b2c5ea598ed9f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:04:50.586 INFO 4486 --- [ocp-schedules-29,10e2a5edef5ce5ae,7d4b2c5ea598ed9f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:04:50.587 INFO 4486 --- [ocp-schedules-29,10e2a5edef5ce5ae,7d4b2c5ea598ed9f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:04:50.587 INFO 4486 --- [ocp-schedules-29,10e2a5edef5ce5ae,7d4b2c5ea598ed9f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:04:50.588 INFO 4486 --- [ocp-schedules-29,10e2a5edef5ce5ae,7d4b2c5ea598ed9f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:04:50.588 INFO 4486 --- [ocp-schedules-29,10e2a5edef5ce5ae,7d4b2c5ea598ed9f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 14:04:51.400 INFO 4486 --- [ocp-schedules-18,5794e57d34448167,2bda42b5a3a16c70] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:04:51.430 INFO 4486 --- [ocp-async-10,d2a7e37005c3dbb7,ade3f9c380a2d816] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:51.690 INFO 4486 --- [ocp-schedules-15,f05084ea71486854,4aa404db182e0e87] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:04:51.690 INFO 4486 --- [ocp-schedules-15,f05084ea71486854,4aa404db182e0e87] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:04:51.690 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:04:51.690 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:04:51.690 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:04:51.690 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:04:51.690 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:04:51.696 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:04:51.696 INFO 4486 --- [ocp-schedules-15,f05084ea71486854,4aa404db182e0e87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:04:51.696 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:04:51.696 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:04:51.696 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:04:51.697 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:04:51.697 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:04:51.698 INFO 4486 --- [ocp-schedules-15,f05084ea71486854,4aa404db182e0e87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:04:51.698 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:04:51.698 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:04:51.698 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:04:51.698 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:04:51.721 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:04:51.723 INFO 4486 --- [ocp-schedules-15,f05084ea71486854,4aa404db182e0e87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:04:51.724 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:04:51.724 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:04:51.724 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:04:51.726 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:04:51.731 INFO 4486 --- [ocp-schedules-15,f05084ea71486854,4aa404db182e0e87] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:04:52.066 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,919f038fb9e966d2,1fa9f74d4004d441] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=919f038fb9e966d2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:52.067 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,919f038fb9e966d2,1fa9f74d4004d441] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=919f038fb9e966d2, duration=2 ms] 2024-10-16 14:04:52.359 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5668c04056ff1677,9302d2b8bbdc01cf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5668c04056ff1677, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:52.359 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,5668c04056ff1677,9302d2b8bbdc01cf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5668c04056ff1677, duration=1 ms] 2024-10-16 14:04:52.588 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:52.994 INFO 4486 --- [prometheus-minute-schedule-1,935867f0cc37d29d,7dc291fa17eb69b3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:53.001 INFO 4486 --- [prometheus-minute-schedule-1,935867f0cc37d29d,7dc291fa17eb69b3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:53.033 INFO 4486 --- [ocp-async-1,0b8f3189962177bc,c8f87796ed4cd801] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:04:53.430 INFO 4486 --- [ocp-async-3,9fa9e23b49819a14,5b7ad157ea6c3e21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:53.591 INFO 4486 --- [ocp-schedules-13,6cd9c429c9e8513b,41722bbbd9719d99] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:04:55.431 INFO 4486 --- [ocp-async-4,5706c20f95ce9f28,9c1c9c5091f4debf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:55.589 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:55.633 INFO 4486 --- [ocp-schedules-12,8dff73a826347cbe,d038330688aca60e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:04:55.656 WARN 4486 --- [scanner-13328109-1,8dff73a826347cbe,c23e09fbec1ca058] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:04:55.729 INFO 4486 --- [alarm-detect-20,8dff73a826347cbe,5dbf8250dd1af15e] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058670 ~ 1729058685 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058685, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058685, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058685=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:04:55.994 INFO 4486 --- [prometheus-minute-schedule-1,4a71276a7cefdea8,bb144d52c25359b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:55.999 INFO 4486 --- [prometheus-minute-schedule-1,4a71276a7cefdea8,bb144d52c25359b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:57.431 INFO 4486 --- [ocp-async-5,70a2ee1980893a18,68927b409ede7c5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:04:58.522 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d4c3ccb6fc2e516a,468b095f15ae4684] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d4c3ccb6fc2e516a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:04:58.523 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d4c3ccb6fc2e516a,468b095f15ae4684] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d4c3ccb6fc2e516a, duration=3 ms] 2024-10-16 14:04:58.590 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:04:58.994 INFO 4486 --- [prometheus-minute-schedule-1,c97740df7d4b3ba3,7c140b0c642c0857] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058700, interval=60 2024-10-16 14:04:59.002 INFO 4486 --- [prometheus-minute-schedule-1,c97740df7d4b3ba3,7c140b0c642c0857] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:04:59.313 INFO 4486 --- [ocp-schedules-8,4cea4d11c4dbd144,6072a99d14f170b5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:04:59.431 INFO 4486 --- [ocp-async-6,16030d7db6c62e44,6db84a67e64ff9df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:00.537 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8596fbf8cf940437,4b7bf5618b86b0b9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8596fbf8cf940437, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:00.538 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,8596fbf8cf940437,4b7bf5618b86b0b9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8596fbf8cf940437, duration=1 ms] 2024-10-16 14:05:01.290 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,682013824fbea105,88b73afb3af1ccb8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=682013824fbea105, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 14:05:01.406 INFO 4486 --- [ocp-schedules-7,24d418a563dc2a74,fb6655516cb26308] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:05:01.431 INFO 4486 --- [ocp-async-9,687a889cd08867e4,c66003096fd58f97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:01.500 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,682013824fbea105,88b73afb3af1ccb8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=682013824fbea105, duration=222 ms] 2024-10-16 14:05:01.578 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,f95ae937ed93d49b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934319, operation:EXECUTE, state: RUNNING 2024-10-16 14:05:01.590 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7934319, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266499, task_operation=execute, latest_execution_start_time=2024-10-16T14:05:01.567+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=7934319}, listMap={}}, executor=10.197.49.184 2024-10-16 14:05:01.590 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:01.628 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:01.690 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:01.896 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:01.936 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:01.947 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for OceanBase 2024-10-16 14:05:01.947 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.ocp.diagnosis.model.NeuralNetwork : W:[999.6945755151274], b:[-1.7128249860997754E12] 2024-10-16 14:05:01.951 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:01.994 INFO 4486 --- [prometheus-minute-schedule-1,1f1bea0b5344a7cf,491003f10521adbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:01.999 INFO 4486 --- [prometheus-minute-schedule-1,1f1bea0b5344a7cf,491003f10521adbe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:02.030 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:02.067 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:02.070 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cc4fbab01742dfcb,b3ed222e23bdbc28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cc4fbab01742dfcb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:02.071 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,cc4fbab01742dfcb,b3ed222e23bdbc28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cc4fbab01742dfcb, duration=1 ms] 2024-10-16 14:05:02.094 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:OceanBase 2024-10-16 14:05:02.168 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.212 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.364 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6e642c5e3e58e4b1,c1347680dcab8177] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6e642c5e3e58e4b1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:02.364 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6e642c5e3e58e4b1,c1347680dcab8177] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6e642c5e3e58e4b1, duration=1 ms] 2024-10-16 14:05:02.462 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.497 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.545 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for obce_3zones 2024-10-16 14:05:02.545 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1437.0635791927157], b:[2.531038719703609E12] 2024-10-16 14:05:02.551 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.704 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.735 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.762 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:obce_3zones 2024-10-16 14:05:02.890 INFO 4486 --- [ocp-schedules-2,a0e9270deba2c91c,3fe6688ab6b095f9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:05:02.890+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:05:02.894 INFO 4486 --- [ocp-schedules-2,a0e9270deba2c91c,3fe6688ab6b095f9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:05:02.901 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:02.908 INFO 4486 --- [ocp-schedules-2,a0e9270deba2c91c,3fe6688ab6b095f9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:05:02.919 INFO 4486 --- [ocp-schedules-2,a0e9270deba2c91c,3fe6688ab6b095f9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:05:02.939 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:03.033 INFO 4486 --- [ocp-async-6,e0a6ac21e8916ece,319ef00a35f5fb13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:05:03.109 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:03.139 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:03.158 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for glob_3zone 2024-10-16 14:05:03.159 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.ocp.diagnosis.model.NeuralNetwork : W:[-1460.824801303744], b:[2.57258492192251E12] 2024-10-16 14:05:03.163 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:03.266 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:03.297 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:03.325 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:glob_3zone 2024-10-16 14:05:03.426 INFO 4486 --- [subtask-executor32,2233273729644a0c,0307fdd1c32dd3fe] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934319, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 14:05:03.431 INFO 4486 --- [ocp-async-10,d36aa2354d4da9c5,a7bb0cfd6efd092e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:04.592 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:04.994 INFO 4486 --- [prometheus-minute-schedule-1,795b794926e9e0da,a419e10bfd69e668] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:05.001 INFO 4486 --- [prometheus-minute-schedule-1,795b794926e9e0da,a419e10bfd69e668] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:05.432 INFO 4486 --- [ocp-async-8,1621ac48225d3e68,1b6730a27bea099b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:05.770 INFO 4486 --- [ocp-schedules-14,d0fed45c1991eb37,49c6da43fcbd4534] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:05:05.811 WARN 4486 --- [scanner-13328651-1,d0fed45c1991eb37,4d39f90a47f72e63] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:05:05.874 INFO 4486 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:5266499, status:SUCCESS 2024-10-16 14:05:05.879 INFO 4486 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:5266499, status:SUCCESS 2024-10-16 14:05:07.066 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:05:07.067 INFO 4486 --- [ocp-schedules-21,49ba7deed0c02eb1,33889578ce8017ac] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:05:07.071 INFO 4486 --- [ocp-schedules-21,49ba7deed0c02eb1,33889578ce8017ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:07.081 INFO 4486 --- [ocp-schedules-10,2313daae2cc42e8f,6f3a7083ede2ab68] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:05:07.083 INFO 4486 --- [ocp-schedules-10,2313daae2cc42e8f,6f3a7083ede2ab68] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:05:07.091 INFO 4486 --- [ocp-schedules-15,bf942057f137246f,97f2661370a1303b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:05:07.091 INFO 4486 --- [ocp-schedules-11,06e70bfd78f84172,d734f3fa393fa9c7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:05:07.092 INFO 4486 --- [host-check-agent-status1,bf942057f137246f,b1272ab137e7475d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:05:07.092 INFO 4486 --- [host-check-agent-status2,bf942057f137246f,dc8c9906dd5076d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:05:07.092 INFO 4486 --- [host-check-clockdiff1,06e70bfd78f84172,8ce30856191af284] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:05:07.092 INFO 4486 --- [ocp-schedules-4,8d3a96613c055a1e,0c73e50080c40aba] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:05:07.092 INFO 4486 --- [host-check-clockdiff2,06e70bfd78f84172,c0198f7c0aec46bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:05:07.092 INFO 4486 --- [host-check-agent-status3,bf942057f137246f,5930f0d5974a5263] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:05:07.093 INFO 4486 --- [host-check-agent-status4,bf942057f137246f,94e12abdb290ce36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:05:07.093 INFO 4486 --- [host-check-clockdiff3,06e70bfd78f84172,0b478eb70c68fdeb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:05:07.094 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:05:07.094 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:05:07.094 INFO 4486 --- [host-check-clockdiff4,06e70bfd78f84172,25ff4c853f28c1c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:05:07.094 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:05:07.094 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:05:07.105 INFO 4486 --- [host-check-agent-status3,bf942057f137246f,5930f0d5974a5263] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:05:07.113 INFO 4486 --- [host-check-agent-status4,bf942057f137246f,94e12abdb290ce36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:05:07.113 INFO 4486 --- [host-check-agent-status2,bf942057f137246f,dc8c9906dd5076d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:05:07.113 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:07.113 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:05:07.117 INFO 4486 --- [host-check-agent-status1,bf942057f137246f,b1272ab137e7475d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:07.117 INFO 4486 --- [host-check-agent-status2,bf942057f137246f,dc8c9906dd5076d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:05:07.117 INFO 4486 --- [host-check-agent-status4,bf942057f137246f,94e12abdb290ce36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:05:07.117 INFO 4486 --- [host-check-agent-status3,bf942057f137246f,5930f0d5974a5263] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:05:07.117 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:07.122 INFO 4486 --- [host-check-clockdiff1,06e70bfd78f84172,8ce30856191af284] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058707 0 0 , error= 2024-10-16 14:05:07.124 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:05:07.124 INFO 4486 --- [host-check-agent-status1,bf942057f137246f,b1272ab137e7475d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:07.124 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:05:07.127 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:05:07.127 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:05:07.129 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:05:07.133 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.133 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:07.134 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:05:07.154 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.155 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:07.155 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:05:07.155 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:05:07.155 INFO 4486 --- [host-check-agent-status2,bf942057f137246f,dc8c9906dd5076d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:05:07.155 INFO 4486 --- [host-check-agent-status4,bf942057f137246f,94e12abdb290ce36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:05:07.155 INFO 4486 --- [host-check-agent-status1,bf942057f137246f,b1272ab137e7475d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:05:07.156 INFO 4486 --- [host-check-agent-status3,bf942057f137246f,5930f0d5974a5263] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:05:07.157 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:05:07.158 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:05:07.161395901+08:00 2024-10-16 14:05:07.159 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:05:07.162636801+08:00 2024-10-16 14:05:07.159 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:05:07.159 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:05:07.158+08:00 2024-10-16 14:05:07.159 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:05:07.159+08:00 2024-10-16 14:05:07.159 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:05:07.159218186+08:00 2024-10-16 14:05:07.160 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:05:07.159+08:00 2024-10-16 14:05:07.161 INFO 4486 --- [host-check-agent-status3,bf942057f137246f,5930f0d5974a5263] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:05:07.161 INFO 4486 --- [host-check-agent-status3,bf942057f137246f,5930f0d5974a5263] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:05:07.161 INFO 4486 --- [host-check-agent-status3,bf942057f137246f,5930f0d5974a5263] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-agent-status4,bf942057f137246f,94e12abdb290ce36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-agent-status4,bf942057f137246f,94e12abdb290ce36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-agent-status4,bf942057f137246f,94e12abdb290ce36] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:05:07.158+08:00 and status AVAILABLE 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:05:07.159+08:00 and status AVAILABLE 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-agent-status1,bf942057f137246f,b1272ab137e7475d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-agent-status1,bf942057f137246f,b1272ab137e7475d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-agent-status1,bf942057f137246f,b1272ab137e7475d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:05:07.162 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:05:07.159+08:00 and status AVAILABLE 2024-10-16 14:05:07.163 INFO 4486 --- [host-check-agent-status2,bf942057f137246f,dc8c9906dd5076d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:05:07.163 INFO 4486 --- [host-check-agent-status2,bf942057f137246f,dc8c9906dd5076d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:05:07.163 INFO 4486 --- [host-check-agent-status2,bf942057f137246f,dc8c9906dd5076d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:05:07.164 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:05:07.164 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:05:07.164 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:05:07.165 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:05:07.166 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:05:07.170168794+08:00 2024-10-16 14:05:07.167 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:05:07.166+08:00 2024-10-16 14:05:07.170 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:05:07.166+08:00 and status AVAILABLE 2024-10-16 14:05:07.171 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.172 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:05:07.172 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:05:07.173 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:07.173 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:05:07.174 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:07.174 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:05:07.175 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:05:07.176 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:05:07.181 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:05:07.182 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:05:07.184 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.185 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:05:07.196 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.197 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:05:07.197 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:07.198 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:07.204 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:07.206 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.207 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:05:07.208 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:07.210 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.212 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:05:07.228 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:07.228 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:05:07.229 INFO 4486 --- [ocp-schedules-26,6cbaf146d666e73a,9dac53f16ded1e0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:05:07.238 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:05:07.239 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:05:07.247 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:05:07.249 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:05:07.256 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:07.257 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:05:07.266 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:05:07.268 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:07.275 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:07.277 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:05:07.291 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:05:07.292 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:07.301 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:05:07.301 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:05:07.312 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:05:07.320 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:07.325 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:05:07.331 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:07.337 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:05:07.347 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:05:07.348 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:05:07.352 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:07.358 INFO 4486 --- [host-check-host-status3,8d3a96613c055a1e,481f9bd42ef09070] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:05:07.358 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:05:07.372 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:07.376 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:05:07.386 INFO 4486 --- [host-check-host-status4,8d3a96613c055a1e,ea01252f774c4d84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:05:07.389 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:05:07.399 INFO 4486 --- [host-check-host-status2,8d3a96613c055a1e,d636562e8a7ea74c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:05:07.429 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:05:07.432 INFO 4486 --- [ocp-async-7,e90bb4cfacfaafd3,8edc24cebcde450f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:07.439 INFO 4486 --- [host-check-host-status1,8d3a96613c055a1e,4f04601b32d18236] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:05:07.592 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:07.666 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c0db3b2ed17d7c72,8ebe9a24b8f4176b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=c0db3b2ed17d7c72, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:05:07.667 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,c0db3b2ed17d7c72,8ebe9a24b8f4176b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=c0db3b2ed17d7c72, duration=3 ms] 2024-10-16 14:05:07.994 INFO 4486 --- [prometheus-minute-schedule-1,690156fe8d6893e1,199bac10ea6955b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:08.001 INFO 4486 --- [prometheus-minute-schedule-1,690156fe8d6893e1,199bac10ea6955b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:08.070 INFO 4486 --- [host-check-clockdiff2,06e70bfd78f84172,c0198f7c0aec46bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058708 4 4 , error= 2024-10-16 14:05:08.528 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ddc6a66ec3ec96c6,83dca843aa2010d9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ddc6a66ec3ec96c6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:08.529 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,ddc6a66ec3ec96c6,83dca843aa2010d9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ddc6a66ec3ec96c6, duration=2 ms] 2024-10-16 14:05:09.432 INFO 4486 --- [ocp-async-4,dff3f4b3e4fa77cd,4c22af35cfb9bb09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:09.644 INFO 4486 --- [host-check-clockdiff4,06e70bfd78f84172,25ff4c853f28c1c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058709 4 4 , error= 2024-10-16 14:05:10.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:05:10.007 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:05:10.024 INFO 4486 --- [Check obproxy1,c1365b3af318b03e,caa62632ca346ed7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:05:10.025 INFO 4486 --- [Check obproxy1,c1365b3af318b03e,caa62632ca346ed7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:10.026 INFO 4486 --- [Check obproxy2,3908f4317ad05564,9aad96c45eb1bde5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:05:10.026 INFO 4486 --- [Check obproxy2,3908f4317ad05564,9aad96c45eb1bde5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:10.032 INFO 4486 --- [Collect obproxy2,654778f4f73884b3,7c54530c9c5ffb0d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:05:10.032 INFO 4486 --- [Collect obproxy1,433080069ef27dd8,b93298d5cce9848e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:05:10.032 INFO 4486 --- [Collect obproxy2,654778f4f73884b3,7c54530c9c5ffb0d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:10.032 INFO 4486 --- [Collect obproxy1,433080069ef27dd8,b93298d5cce9848e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:10.047 INFO 4486 --- [Check obproxy1,c1365b3af318b03e,caa62632ca346ed7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:05:10.047 INFO 4486 --- [Check obproxy1,c1365b3af318b03e,caa62632ca346ed7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:05:10.050 INFO 4486 --- [Check obproxy2,3908f4317ad05564,9aad96c45eb1bde5] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:05:10.050 INFO 4486 --- [Check obproxy2,3908f4317ad05564,9aad96c45eb1bde5] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:05:10.051 INFO 4486 --- [Collect obproxy1,433080069ef27dd8,b93298d5cce9848e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:05:10.051 INFO 4486 --- [Collect obproxy2,654778f4f73884b3,7c54530c9c5ffb0d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:05:10.053 INFO 4486 --- [Collect obproxy1,433080069ef27dd8,b93298d5cce9848e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:05:10.056 INFO 4486 --- [Collect obproxy1,433080069ef27dd8,b93298d5cce9848e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:05:10.057 INFO 4486 --- [Collect obproxy2,654778f4f73884b3,7c54530c9c5ffb0d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:05:10.061 INFO 4486 --- [Collect obproxy2,654778f4f73884b3,7c54530c9c5ffb0d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:05:10.269 INFO 4486 --- [host-check-clockdiff3,06e70bfd78f84172,0b478eb70c68fdeb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058710 4 4 , error= 2024-10-16 14:05:10.542 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1d383f1d2073b58d,d871ddd2d314e5fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1d383f1d2073b58d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:10.542 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1d383f1d2073b58d,d871ddd2d314e5fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1d383f1d2073b58d, duration=1 ms] 2024-10-16 14:05:10.593 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:05:10.994 INFO 4486 --- [prometheus-minute-schedule-1,90f9ce99b6c1898e,ee5c385ea76caa7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:10.999 INFO 4486 --- [prometheus-minute-schedule-1,90f9ce99b6c1898e,ee5c385ea76caa7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:05:11.411 INFO 4486 --- [ocp-schedules-5,ddb515fba9fe15cd,d44112aebcc9549a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:05:11.432 INFO 4486 --- [ocp-async-2,b94d106ca2998530,a4b30868e54c6d4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:11.749 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9d52bba71a2cb2f8,06e28c456c88d943] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=9d52bba71a2cb2f8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:05:11.749 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,9d52bba71a2cb2f8,06e28c456c88d943] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=9d52bba71a2cb2f8, duration=1 ms] 2024-10-16 14:05:12.080 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,44cb4597f413eed7,9fc135656d966dd2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=44cb4597f413eed7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:12.080 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,44cb4597f413eed7,9fc135656d966dd2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=44cb4597f413eed7, duration=3 ms] 2024-10-16 14:05:12.368 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,dcae51367a8322cf,f8ba10e11787901b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=dcae51367a8322cf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:12.368 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,dcae51367a8322cf,f8ba10e11787901b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=dcae51367a8322cf, duration=1 ms] 2024-10-16 14:05:13.033 INFO 4486 --- [ocp-async-10,644143611444d9c9,1287ba7a53bfcef4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:05:13.433 INFO 4486 --- [ocp-async-3,ced40603d2a9650c,8e318dd77330ffd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:13.594 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:13.994 INFO 4486 --- [prometheus-minute-schedule-1,8690956f07dd91c7,1fef9cb2ca0b3d72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:14.000 INFO 4486 --- [prometheus-minute-schedule-1,8690956f07dd91c7,1fef9cb2ca0b3d72] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:14.704 INFO 4486 --- [ocp-schedules-16,e44a539ff847f11f,4bb0fab6a8e58759] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:05:14.704 INFO 4486 --- [ocp-schedules-16,c1b2aec2f55f096c,9cf6472ae45d2ea8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:05:14.706 ERROR 4486 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=835727) Duplicate entry '669741696490737664-1729058640' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:05:14.704513] [YB420AC531B8-00060B7DB4196DDE-0-0]; nested exception is java.sql.BatchUpdateException: (conn=835727) Duplicate entry '669741696490737664-1729058640' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:05:14.704513] [YB420AC531B8-00060B7DB4196DDE-0-0] 2024-10-16 14:05:14.707 INFO 4486 --- [ocp-schedules-16,c1b2aec2f55f096c,9cf6472ae45d2ea8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:14.713 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058640, nodeCount=1000, affectRows=1000 2024-10-16 14:05:14.715 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058640, nodeCount=1000, affectRows=1000 2024-10-16 14:05:14.718 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058640, nodeCount=1000, validCount=1000 2024-10-16 14:05:14.720 INFO 4486 --- [metric-archive-second-13,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058640, nodeCount=999, validCount=999 2024-10-16 14:05:14.733 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058640, nodeCount=1000, affectRows=1000 2024-10-16 14:05:14.738 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058640, nodeCount=1000, validCount=1000 2024-10-16 14:05:14.740 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.741 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:14.750 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.751 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:14.777 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.778 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:14.783 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:14.785 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.785 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:05:14.820 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:14.821 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.822 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:14.823 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:05:14.824 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.828 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:05:14.829 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:14.840 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:14.841 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:14.856 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:14.868 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:14.868 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.869 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:14.870 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:05:14.876 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:05:14.883 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:14.886 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:14.894 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:14.896 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.897 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:05:14.916 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:05:14.955 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:14.957 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.957 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:14.958 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:05:14.958 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:05:14.961 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:14.964 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:05:14.968 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:14.986 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058640, nodeCount=451, affectRows=451 2024-10-16 14:05:14.989 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058640, nodeCount=449, validCount=449 2024-10-16 14:05:15.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:15.001 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:15.004 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:05:15.015 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:15.016 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:15.016 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:05:15.028 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:05:15.028 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:05:15.033 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:15.033 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:15.064 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:15.064 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:15.096 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:15.096 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:15.122 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:15.123 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:15.152 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:15.182 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:15.184 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:15.185 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:15.189 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:05:15.195 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:05:15.199 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:05:15.201 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:05:15.221 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:05:15.297 INFO 4486 --- [metric-archive-async-27,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058640, nodeCount=74, affectRows=74 2024-10-16 14:05:15.298 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058640, nodeCount=92, affectRows=92 2024-10-16 14:05:15.306 INFO 4486 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058640, nodeCount=147, affectRows=147 2024-10-16 14:05:15.308 INFO 4486 --- [metric-archive-async-10,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058640, nodeCount=167, affectRows=167 2024-10-16 14:05:15.356 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:05:15.386 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:15.388 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:15.389 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:05:15.422 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:05:15.422 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:05:15.423 INFO 4486 --- [metric-archive-async-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058640, nodeCount=74, validCount=74 2024-10-16 14:05:15.433 INFO 4486 --- [ocp-async-4,3195965d00802a61,7af7fa829a49e6df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:15.434 INFO 4486 --- [metric-archive-async-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058640, nodeCount=92, validCount=92 2024-10-16 14:05:15.529 INFO 4486 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058640, nodeCount=147, validCount=147 2024-10-16 14:05:15.556 INFO 4486 --- [metric-archive-async-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058640, nodeCount=167, validCount=167 2024-10-16 14:05:16.021 INFO 4486 --- [ocp-schedules-27,c47999dd10165b88,81df94d97347f886] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:05:16.054 WARN 4486 --- [scanner-13329815-1,c47999dd10165b88,8903cac979b0ef12] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:05:16.122 INFO 4486 --- [alarm-detect-14,c47999dd10165b88,69ce44a73b68ab4d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058685 ~ 1729058706 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058700, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058700, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058700=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:05:16.595 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:16.994 INFO 4486 --- [prometheus-minute-schedule-1,54bafdac09dcd1d6,b497ae55d458ce0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:17.001 INFO 4486 --- [prometheus-minute-schedule-1,54bafdac09dcd1d6,b497ae55d458ce0f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:17.009 INFO 4486 --- [ocp-schedules-17,6ac943ff27b3141c,dfb00414940000a3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 14:05:17.009 INFO 4486 --- [ocp-schedules-17,6ac943ff27b3141c,dfb00414940000a3] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 14:05:17.079 INFO 4486 --- [ocp-schedules-21,49ba7deed0c02eb1,33889578ce8017ac] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:05:17.099 INFO 4486 --- [pool-284-thread-2,6ac943ff27b3141c,46dd1c55daa022e0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:05:17.099 INFO 4486 --- [pool-284-thread-1,6ac943ff27b3141c,83644dbf50808a84] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:05:17.099 INFO 4486 --- [pool-284-thread-3,6ac943ff27b3141c,2128ef7ff6a329aa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:05:17.124 INFO 4486 --- [pool-284-thread-1,6ac943ff27b3141c,83644dbf50808a84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:17.126 INFO 4486 --- [pool-284-thread-2,6ac943ff27b3141c,46dd1c55daa022e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:17.128 INFO 4486 --- [pool-284-thread-1,6ac943ff27b3141c,83644dbf50808a84] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058717000000] 2024-10-16 14:05:17.130 INFO 4486 --- [pool-284-thread-2,6ac943ff27b3141c,46dd1c55daa022e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058717000000] 2024-10-16 14:05:17.134 INFO 4486 --- [pool-284-thread-3,6ac943ff27b3141c,2128ef7ff6a329aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:17.138 INFO 4486 --- [pool-284-thread-3,6ac943ff27b3141c,2128ef7ff6a329aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729058717000000] 2024-10-16 14:05:17.159 INFO 4486 --- [ocp-schedules-10,2313daae2cc42e8f,6f3a7083ede2ab68] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:05:17.304 INFO 4486 --- [ocp-schedules-12,7edc4427455fe2d2,f60e5a6f047d328f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:05:17.415 WARN 4486 --- [pool-284-thread-3,6ac943ff27b3141c,2128ef7ff6a329aa] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 14:05:17.415 WARN 4486 --- [pool-284-thread-3,6ac943ff27b3141c,2128ef7ff6a329aa] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 14:05:17.417 INFO 4486 --- [pool-284-thread-3,6ac943ff27b3141c,2128ef7ff6a329aa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:05:17.433 INFO 4486 --- [ocp-async-2,98268b218eb1d1a8,bbb5c417b991d8e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:17.744 WARN 4486 --- [pool-284-thread-1,6ac943ff27b3141c,83644dbf50808a84] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 14:05:17.745 WARN 4486 --- [pool-284-thread-1,6ac943ff27b3141c,83644dbf50808a84] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1009 2024-10-16 14:05:17.746 INFO 4486 --- [pool-284-thread-1,6ac943ff27b3141c,83644dbf50808a84] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:05:17.957 WARN 4486 --- [pool-284-thread-2,6ac943ff27b3141c,46dd1c55daa022e0] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 14:05:17.959 INFO 4486 --- [pool-284-thread-2,6ac943ff27b3141c,46dd1c55daa022e0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:05:17.961 INFO 4486 --- [ocp-schedules-17,6ac943ff27b3141c,dfb00414940000a3] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 14:05:18.333 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 14:05:18.333 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 14:05:18.341 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 14:05:18.341 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 14:05:18.341 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 14:05:18.341 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 14:05:18.347 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 14:05:18.347 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 14:05:18.347 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 14:05:18.347 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 14:05:18.396 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 14:05:18.403 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 14:05:18.404 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 14:05:18.404 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 14:05:18.404 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 14:05:18.404 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 14:05:18.411 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 14:05:18.534 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b658a76b76e9ca5d,767f2851770c5409] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b658a76b76e9ca5d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:18.534 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b658a76b76e9ca5d,767f2851770c5409] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b658a76b76e9ca5d, duration=2 ms] 2024-10-16 14:05:18.916 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 14:05:19.319 INFO 4486 --- [ocp-schedules-5,8a8e7eb142c57d61,905a00842f766416] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:05:19.433 INFO 4486 --- [ocp-async-7,3258b78f3e19552c,666d57df5bd84a9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:19.442 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.095076667S 2024-10-16 14:05:19.442 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 14:05:19.536 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 14:05:19.536 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 14:05:19.562 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 14:05:19.637 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 14:05:19.637 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 14:05:19.637 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 14:05:19.637 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 14:05:19.637 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 14:05:19.638 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 14:05:19.647 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 14:05:19.829 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] o.s.boot.SpringApplication : Started application in 1.488 seconds (JVM running for 79622.88) 2024-10-16 14:05:19.829 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.48897062S 2024-10-16 14:05:19.994 INFO 4486 --- [prometheus-minute-schedule-1,3ac9b0dd82946bfa,7b7195a5ee43c29d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:20.000 INFO 4486 --- [prometheus-minute-schedule-1,3ac9b0dd82946bfa,7b7195a5ee43c29d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:20.160 INFO 4486 --- [ocp-schedules-1,62acd0dd23a33502,e444504f84d17ab6] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 14:05:20.547 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,fb915266ac235c3d,24001d1bf1b3917c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fb915266ac235c3d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:20.547 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,fb915266ac235c3d,24001d1bf1b3917c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fb915266ac235c3d, duration=1 ms] 2024-10-16 14:05:20.996 INFO 4486 --- [prometheus-cache-monitor1,759bef18988a3553,91b42d00920aa95a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8131} 2024-10-16 14:05:20.996 INFO 4486 --- [prometheus-cache-monitor1,759bef18988a3553,91b42d00920aa95a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:05:21.418 INFO 4486 --- [ocp-schedules-24,0da2d995582ed5dd,1cce17ab0c53fbab] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:05:21.433 INFO 4486 --- [ocp-async-1,13f291665635a1a8,1e8eda5953735f03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:22.085 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8a01c609cd6f4867,072ae0a75bd4d05c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8a01c609cd6f4867, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:22.085 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8a01c609cd6f4867,072ae0a75bd4d05c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8a01c609cd6f4867, duration=1 ms] 2024-10-16 14:05:22.373 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6306003ac2a37254,4eb1534b70b7781f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6306003ac2a37254, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:22.373 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6306003ac2a37254,4eb1534b70b7781f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6306003ac2a37254, duration=1 ms] 2024-10-16 14:05:22.597 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:22.994 INFO 4486 --- [prometheus-minute-schedule-1,b87b37a75649c9a6,535fcb42b8c70821] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:23.000 INFO 4486 --- [prometheus-minute-schedule-1,b87b37a75649c9a6,535fcb42b8c70821] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:23.033 INFO 4486 --- [ocp-async-7,fb692a628f54c226,8d779b6d66fea164] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:05:23.434 INFO 4486 --- [ocp-async-10,2403457581313033,52f69723c1ab7386] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:24.122 INFO 4486 --- [ocp-schedules-11,3e54a688ce463885,83905eb49cd7e4fd] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:05:25.352 INFO 4486 --- [ocp-schedules-14,a4bc18e83faf2971,b868f3134ffa7de5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:05:25.434 INFO 4486 --- [ocp-async-8,02eeacb1d217136a,14702c643fbc2ffe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:25.598 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:25.994 INFO 4486 --- [prometheus-minute-schedule-1,bdd6673e0ed0b8c3,4cd2b5c16d7d4d8f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:26.000 INFO 4486 --- [prometheus-minute-schedule-1,bdd6673e0ed0b8c3,4cd2b5c16d7d4d8f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:26.171 INFO 4486 --- [ocp-schedules-12,0964eb77f692385f,ee401cea916bd8bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:05:26.207 WARN 4486 --- [scanner-13330064-1,0964eb77f692385f,a5c5a54705e70ab8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:05:26.254 INFO 4486 --- [alarm-detect-12,0964eb77f692385f,fe438bc2a1ddb62f] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058700 ~ 1729058716 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058715, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058715, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058715=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:05:27.072 INFO 4486 --- [ocp-schedules-14,632a5fe313303ba6,2e9a71cf23a450d0] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:05:27.073 INFO 4486 --- [ocp-schedules-14,632a5fe313303ba6,2e9a71cf23a450d0] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:05:27.104 INFO 4486 --- [Collect all tenants major compactions1,632a5fe313303ba6,2b1813c559e858f8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:05:27.104 INFO 4486 --- [Collect all tenants major compactions2,632a5fe313303ba6,ef3fd3bdd3505296] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:05:27.104 INFO 4486 --- [Collect all tenants major compactions3,632a5fe313303ba6,62fd595c6c5e3bf3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:05:27.105 INFO 4486 --- [Collect all tenants major compactions4,632a5fe313303ba6,e1a28a002d8c551f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:05:27.105 INFO 4486 --- [Collect all tenants major compactions5,632a5fe313303ba6,c62773ac98e057d1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:05:27.120 INFO 4486 --- [Collect all tenants major compactions1,632a5fe313303ba6,2b1813c559e858f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:27.121 INFO 4486 --- [Collect all tenants major compactions3,632a5fe313303ba6,62fd595c6c5e3bf3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:27.121 INFO 4486 --- [Collect all tenants major compactions2,632a5fe313303ba6,ef3fd3bdd3505296] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:27.121 INFO 4486 --- [Collect all tenants major compactions1,632a5fe313303ba6,2b1813c559e858f8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:05:27.123 INFO 4486 --- [Collect all tenants major compactions3,632a5fe313303ba6,62fd595c6c5e3bf3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:05:27.123 INFO 4486 --- [Collect all tenants major compactions2,632a5fe313303ba6,ef3fd3bdd3505296] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:05:27.124 INFO 4486 --- [Collect all tenants major compactions1,632a5fe313303ba6,2b1813c559e858f8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:05:27.126 INFO 4486 --- [Collect all tenants major compactions2,632a5fe313303ba6,ef3fd3bdd3505296] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:05:27.126 INFO 4486 --- [Collect all tenants major compactions3,632a5fe313303ba6,62fd595c6c5e3bf3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:05:27.128 INFO 4486 --- [Collect all tenants major compactions1,632a5fe313303ba6,2b1813c559e858f8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:05:27.132 INFO 4486 --- [Collect all tenants major compactions2,632a5fe313303ba6,ef3fd3bdd3505296] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:05:27.133 INFO 4486 --- [Collect all tenants major compactions3,632a5fe313303ba6,62fd595c6c5e3bf3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:05:27.134 INFO 4486 --- [Collect all tenants major compactions5,632a5fe313303ba6,c62773ac98e057d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:27.135 INFO 4486 --- [Collect all tenants major compactions4,632a5fe313303ba6,e1a28a002d8c551f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:27.136 INFO 4486 --- [Collect all tenants major compactions5,632a5fe313303ba6,c62773ac98e057d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:05:27.137 INFO 4486 --- [Collect all tenants major compactions4,632a5fe313303ba6,e1a28a002d8c551f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:05:27.141 INFO 4486 --- [Collect all tenants major compactions5,632a5fe313303ba6,c62773ac98e057d1] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:05:27.141 INFO 4486 --- [Collect all tenants major compactions5,632a5fe313303ba6,c62773ac98e057d1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:05:27.141 INFO 4486 --- [Collect all tenants major compactions4,632a5fe313303ba6,e1a28a002d8c551f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:05:27.141 INFO 4486 --- [Collect all tenants major compactions4,632a5fe313303ba6,e1a28a002d8c551f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:05:27.142 INFO 4486 --- [ocp-schedules-14,632a5fe313303ba6,2e9a71cf23a450d0] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:05:27.434 INFO 4486 --- [ocp-async-7,a420130bd0b54fcd,c9a3df221e51942a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:28.007 INFO 4486 --- [ocp-schedules-26,039ae08b75821cfb,65a8f9c8a838a7e5] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:05:28.007+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:05:28.009 INFO 4486 --- [ocp-schedules-26,039ae08b75821cfb,65a8f9c8a838a7e5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:05:28.010 INFO 4486 --- [ocp-schedules-26,039ae08b75821cfb,65a8f9c8a838a7e5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:05:28.010 INFO 4486 --- [ocp-schedules-26,039ae08b75821cfb,65a8f9c8a838a7e5] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:05:28.539 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,64cfffd4ce18cfd1,71bfdb512af74ad5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=64cfffd4ce18cfd1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:28.539 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,64cfffd4ce18cfd1,71bfdb512af74ad5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=64cfffd4ce18cfd1, duration=1 ms] 2024-10-16 14:05:28.598 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:28.994 INFO 4486 --- [prometheus-minute-schedule-1,861825aaa96af9e2,a54da2db4d159b22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:28.999 INFO 4486 --- [prometheus-minute-schedule-1,861825aaa96af9e2,a54da2db4d159b22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:29.434 INFO 4486 --- [ocp-async-4,cb147ca6b6bfca17,a641978132e2c636] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:29.909 INFO 4486 --- [ocp-schedules-3,a1c9115ed4331877,43de38ebe8464e9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 14:05:30.102 INFO 4486 --- [prometheus-second-schedule-1,ad8efef3a3187f82,2c562cda45a591b7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:05:30.553 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,0aefa28fd64081c3,d9aabadd363ba79c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0aefa28fd64081c3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:30.553 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,0aefa28fd64081c3,d9aabadd363ba79c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0aefa28fd64081c3, duration=1 ms] 2024-10-16 14:05:31.423 INFO 4486 --- [ocp-schedules-26,1a8aa60e2bc2308c,0f022fe0901b8588] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:05:31.435 INFO 4486 --- [ocp-async-6,44d19983890a5578,d2acb989b9d7ca33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:31.599 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:31.994 INFO 4486 --- [prometheus-minute-schedule-1,f2669c175b3fffef,7544f33ff90ec629] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:31.999 INFO 4486 --- [prometheus-minute-schedule-1,f2669c175b3fffef,7544f33ff90ec629] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:32.089 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f61e3ba9807f2dc3,6b0f2f2ecb17c148] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f61e3ba9807f2dc3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:32.089 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f61e3ba9807f2dc3,6b0f2f2ecb17c148] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f61e3ba9807f2dc3, duration=1 ms] 2024-10-16 14:05:32.397 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7fe7bf64812d5185,62bec2054f365ba8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7fe7bf64812d5185, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:32.397 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7fe7bf64812d5185,62bec2054f365ba8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7fe7bf64812d5185, duration=1 ms] 2024-10-16 14:05:33.033 INFO 4486 --- [ocp-async-2,bb904fe2aafe89c5,301f89022bc5156b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:05:33.435 INFO 4486 --- [ocp-async-1,c78fc4a7a00c5177,cc2690545ee0dab5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:34.600 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:34.994 INFO 4486 --- [prometheus-minute-schedule-1,135c518966f63cf1,f6840d56a4c60481] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:35.000 INFO 4486 --- [prometheus-minute-schedule-1,135c518966f63cf1,f6840d56a4c60481] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:35.435 INFO 4486 --- [ocp-async-4,47eb76fcd2435294,40a26acc1ec1d8d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:36.334 INFO 4486 --- [ocp-schedules-13,ad3686d441564acc,b363576311f4aca4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:05:36.352 WARN 4486 --- [scanner-13330345-1,ad3686d441564acc,35458b6ee285b267] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:05:36.643 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a226bd2522d7d690,6c32a692408d2ec9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=a226bd2522d7d690, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:05:36.644 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,a226bd2522d7d690,6c32a692408d2ec9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=a226bd2522d7d690, duration=2 ms] 2024-10-16 14:05:37.064 INFO 4486 --- [ocp-schedules-26,9febaa243ab5cfe0,16191889b46765e7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:05:37.064 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:05:37.066 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:05:37.066 INFO 4486 --- [ocp-schedules-27,140518d0e95f77a2,d0278dc7e36b064c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:05:37.068 INFO 4486 --- [ocp-schedules-27,140518d0e95f77a2,d0278dc7e36b064c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:37.069 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:05:37.072 INFO 4486 --- [ocp-schedules-26,9febaa243ab5cfe0,16191889b46765e7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:05:37.081 INFO 4486 --- [ocp-schedules-25,783c21156bb7e916,e530e66ffc08cc7a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:05:37.083 INFO 4486 --- [ocp-schedules-25,783c21156bb7e916,e530e66ffc08cc7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:05:37.090 INFO 4486 --- [ocp-schedules-17,534f95eb5db234d7,96e3efc3024e5b12] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:05:37.090 INFO 4486 --- [ocp-schedules-5,7b41d85d5230281d,a0e5fc0dbd70cb49] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:05:37.091 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:05:37.091 INFO 4486 --- [ocp-schedules-6,0c805be6f42f27a0,9d747e67722bd89e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:05:37.091 INFO 4486 --- [host-check-agent-status1,534f95eb5db234d7,6a7608d1624f1d17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:05:37.091 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:05:37.091 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:05:37.091 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:05:37.092 INFO 4486 --- [host-check-agent-status2,534f95eb5db234d7,383b19d58bdb908b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:05:37.092 INFO 4486 --- [host-check-agent-status3,534f95eb5db234d7,79e65cd36cf46812] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:05:37.093 INFO 4486 --- [host-check-agent-status4,534f95eb5db234d7,c5e5db7f70e80f93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:05:37.093 INFO 4486 --- [host-check-clockdiff1,0c805be6f42f27a0,c02ad701ce2558fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:05:37.094 INFO 4486 --- [host-check-clockdiff2,0c805be6f42f27a0,188e95019fb09d97] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:05:37.094 INFO 4486 --- [host-check-clockdiff3,0c805be6f42f27a0,06df3dc60b708568] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:05:37.094 INFO 4486 --- [host-check-clockdiff4,0c805be6f42f27a0,239955d390ad1d23] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:05:37.103 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:05:37.099 INFO 4486 --- [host-check-agent-status1,534f95eb5db234d7,6a7608d1624f1d17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:37.121 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:05:37.121 INFO 4486 --- [host-check-agent-status3,534f95eb5db234d7,79e65cd36cf46812] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:05:37.121 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:05:37.121 INFO 4486 --- [host-check-agent-status1,534f95eb5db234d7,6a7608d1624f1d17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:37.122 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:05:37.129 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:05:37.129 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:05:37.130 INFO 4486 --- [host-check-agent-status3,534f95eb5db234d7,79e65cd36cf46812] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:05:37.130 INFO 4486 --- [host-check-clockdiff4,0c805be6f42f27a0,239955d390ad1d23] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058737 4 4 , error= 2024-10-16 14:05:37.130 INFO 4486 --- [host-check-clockdiff3,0c805be6f42f27a0,06df3dc60b708568] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058737 4 4 , error= 2024-10-16 14:05:37.130 INFO 4486 --- [host-check-clockdiff2,0c805be6f42f27a0,188e95019fb09d97] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058737 4 4 , error= 2024-10-16 14:05:37.133 INFO 4486 --- [host-check-agent-status4,534f95eb5db234d7,c5e5db7f70e80f93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:05:37.133 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:37.133 INFO 4486 --- [host-check-agent-status2,534f95eb5db234d7,383b19d58bdb908b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:05:37.144 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:37.140 INFO 4486 --- [host-check-clockdiff1,0c805be6f42f27a0,c02ad701ce2558fe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058737 0 0 , error= 2024-10-16 14:05:37.145 INFO 4486 --- [host-check-agent-status4,534f95eb5db234d7,c5e5db7f70e80f93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:05:37.146 INFO 4486 --- [host-check-agent-status2,534f95eb5db234d7,383b19d58bdb908b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:05:37.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.152 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:37.152 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:05:37.170 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:05:37.170 INFO 4486 --- [host-check-agent-status3,534f95eb5db234d7,79e65cd36cf46812] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:05:37.170 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:37.171 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.172 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:05:37.172 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:05:37.172 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:37.173 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:05:37.176 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:05:37.177 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:05:37.174091901+08:00 2024-10-16 14:05:37.177 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:05:37.178032249+08:00 2024-10-16 14:05:37.177 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:05:37.177998698+08:00 2024-10-16 14:05:37.177 INFO 4486 --- [host-check-agent-status3,534f95eb5db234d7,79e65cd36cf46812] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:05:37.177 INFO 4486 --- [host-check-agent-status3,534f95eb5db234d7,79e65cd36cf46812] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:05:37.177 INFO 4486 --- [host-check-agent-status3,534f95eb5db234d7,79e65cd36cf46812] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:05:37.178 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:05:37.177+08:00 2024-10-16 14:05:37.178 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:05:37.177+08:00 2024-10-16 14:05:37.178 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:05:37.177+08:00 2024-10-16 14:05:37.178 INFO 4486 --- [host-check-agent-status1,534f95eb5db234d7,6a7608d1624f1d17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:05:37.178 INFO 4486 --- [host-check-agent-status4,534f95eb5db234d7,c5e5db7f70e80f93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:05:37.178 INFO 4486 --- [host-check-agent-status2,534f95eb5db234d7,383b19d58bdb908b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:05:37.179 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:05:37.182354511+08:00 2024-10-16 14:05:37.180 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:05:37.179+08:00 2024-10-16 14:05:37.181 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.182 INFO 4486 --- [host-check-agent-status2,534f95eb5db234d7,383b19d58bdb908b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:05:37.182 INFO 4486 --- [host-check-agent-status2,534f95eb5db234d7,383b19d58bdb908b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:05:37.182 INFO 4486 --- [host-check-agent-status2,534f95eb5db234d7,383b19d58bdb908b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:05:37.182 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:05:37.182 INFO 4486 --- [host-check-agent-status1,534f95eb5db234d7,6a7608d1624f1d17] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:05:37.182 INFO 4486 --- [host-check-agent-status1,534f95eb5db234d7,6a7608d1624f1d17] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:05:37.182 INFO 4486 --- [host-check-agent-status1,534f95eb5db234d7,6a7608d1624f1d17] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:05:37.184 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.184 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.185 INFO 4486 --- [host-check-agent-status4,534f95eb5db234d7,c5e5db7f70e80f93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:05:37.185 INFO 4486 --- [host-check-agent-status4,534f95eb5db234d7,c5e5db7f70e80f93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:05:37.185 INFO 4486 --- [host-check-agent-status4,534f95eb5db234d7,c5e5db7f70e80f93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:05:37.185 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:05:37.185 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:05:37.177+08:00 and status AVAILABLE 2024-10-16 14:05:37.188 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:05:37.188 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:05:37.177+08:00 and status AVAILABLE 2024-10-16 14:05:37.189 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:05:37.177+08:00 and status AVAILABLE 2024-10-16 14:05:37.191 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:05:37.192 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:05:37.179+08:00 and status AVAILABLE 2024-10-16 14:05:37.192 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:05:37.192 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.194 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:05:37.196 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.196 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.196 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.196 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.196 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.196 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:05:37.196 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.196 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:05:37.197 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:05:37.200 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:05:37.200 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:37.201 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:05:37.202 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:37.204 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:05:37.205 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:05:37.210 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.211 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:05:37.212 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:05:37.219 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:37.220 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.221 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:05:37.224 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:37.225 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:37.230 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:05:37.238 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.239 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:05:37.241 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.242 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:05:37.249 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:05:37.255 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:05:37.257 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.258 INFO 4486 --- [ocp-schedules-8,556faf19550e373e,7d68b5f8346dc6e9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:05:37.261 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:05:37.263 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:05:37.265 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:05:37.274 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:05:37.286 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:05:37.287 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:37.287 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:37.292 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:37.296 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:05:37.311 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:05:37.316 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:05:37.324 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:05:37.325 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:05:37.326 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:05:37.333 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:05:37.340 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:05:37.344 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:37.348 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:37.359 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:37.362 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:05:37.372 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:05:37.375 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:05:37.377 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:05:37.385 INFO 4486 --- [host-check-host-status3,7b41d85d5230281d,987b2f1ebbba5cc4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:05:37.387 INFO 4486 --- [host-check-host-status2,7b41d85d5230281d,3c02169b50aae0dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:05:37.392 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:05:37.393 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:05:37.407 INFO 4486 --- [host-check-host-status4,7b41d85d5230281d,53b5cdc0a49379ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:05:37.415 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:37.417 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.418 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:05:37.419 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.419 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.420 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:05:37.435 INFO 4486 --- [ocp-async-8,b019cd3eb3c22dfa,a6523fc695391e24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:37.435 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.435 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.435 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.435 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.435 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.436 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.451 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:05:37.461 INFO 4486 --- [host-check-host-status1,7b41d85d5230281d,4ccb0aa6f9c85e13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:05:37.467 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:05:37.509 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:05:37.552 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:37.568 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:37.570 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.570 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:05:37.571 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:05:37.577 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.578 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:05:37.579 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.580 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.581 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:05:37.592 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.592 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.592 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.592 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.592 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.593 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.601 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:37.601 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:05:37.790 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:05:37.836 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:37.838 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.841 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:05:37.842 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.842 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.843 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:05:37.855 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.855 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.855 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.855 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.855 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:37.855 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:37.873 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:05:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:05:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:05:37.994 INFO 4486 --- [prometheus-minute-schedule-1,e62c591d03cca1fb,147796761a0977e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:38.000 INFO 4486 --- [prometheus-minute-schedule-1,e62c591d03cca1fb,147796761a0977e5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:38.076 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:05:38.122 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:38.124 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:38.124 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:05:38.125 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:38.127 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:38.127 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:05:38.137 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:38.137 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:38.137 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:38.137 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:38.137 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:05:38.138 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:38.227 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:05:38.442 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:05:38.231464, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:05:38.442 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:05:38.231464, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:05:38.465 INFO 4486 --- [ocp-schedules-11,5f4c3566c08d9037,e8139f012b96a570] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 14:05:38.465 INFO 4486 --- [ocp-schedules-11,5f4c3566c08d9037,e8139f012b96a570] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 14:05:38.466 INFO 4486 --- [ocp-schedules-11,5f4c3566c08d9037,e8139f012b96a570] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 14:05:38.482 INFO 4486 --- [ocp-schedules-8,0d0c38ae0f4138f4,d0017da584a07124] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13798,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3451,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8131} 2024-10-16 14:05:38.483 INFO 4486 --- [ocp-schedules-8,0d0c38ae0f4138f4,d0017da584a07124] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:05:38.483 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:05:38.231464} } 2024-10-16 14:05:38.544 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ee05b9b58aba6def,ec94381b38ae7178] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ee05b9b58aba6def, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:38.544 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,ee05b9b58aba6def,ec94381b38ae7178] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ee05b9b58aba6def, duration=2 ms] 2024-10-16 14:05:38.545 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:05:38.552 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:05:38.555 INFO 4486 --- [ocp-schedules-7,8efd680e2ef94e38,8266f55d909eb575] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:05:39.324 INFO 4486 --- [ocp-schedules-30,8265ef95146385d0,db8c2afe3ce03b49] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:05:39.435 INFO 4486 --- [ocp-async-1,e53eccb0d40afd38,a9c5ea6ce87c4087] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:40.006 INFO 4486 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:05:40.007 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:05:40.027 INFO 4486 --- [Check obproxy1,e20ae3b29fc2f533,cd00527e4203c8c7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:05:40.027 INFO 4486 --- [Check obproxy2,47794d55ce511cfb,d2451388092249d6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:05:40.032 INFO 4486 --- [Collect obproxy2,1efc08b840fc37f1,63c14086113c8d64] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:05:40.032 INFO 4486 --- [Check obproxy2,47794d55ce511cfb,d2451388092249d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:40.032 INFO 4486 --- [Collect obproxy1,43ba03092105164e,a881132a6cc6fadd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:05:40.032 INFO 4486 --- [Collect obproxy1,43ba03092105164e,a881132a6cc6fadd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:40.032 INFO 4486 --- [Collect obproxy2,1efc08b840fc37f1,63c14086113c8d64] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:40.033 INFO 4486 --- [Check obproxy1,e20ae3b29fc2f533,cd00527e4203c8c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:05:40.058 INFO 4486 --- [Collect obproxy1,43ba03092105164e,a881132a6cc6fadd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:05:40.060 INFO 4486 --- [Collect obproxy2,1efc08b840fc37f1,63c14086113c8d64] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:05:40.061 INFO 4486 --- [Check obproxy1,e20ae3b29fc2f533,cd00527e4203c8c7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:05:40.062 INFO 4486 --- [Check obproxy1,e20ae3b29fc2f533,cd00527e4203c8c7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:05:40.062 INFO 4486 --- [Check obproxy2,47794d55ce511cfb,d2451388092249d6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:05:40.062 INFO 4486 --- [Check obproxy2,47794d55ce511cfb,d2451388092249d6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:05:40.063 INFO 4486 --- [Collect obproxy1,43ba03092105164e,a881132a6cc6fadd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:05:40.064 INFO 4486 --- [Collect obproxy2,1efc08b840fc37f1,63c14086113c8d64] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:05:40.067 INFO 4486 --- [Collect obproxy1,43ba03092105164e,a881132a6cc6fadd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:05:40.070 INFO 4486 --- [Collect obproxy2,1efc08b840fc37f1,63c14086113c8d64] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:05:40.562 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,9985865e6944c6a5,78ce420f60cdfb05] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9985865e6944c6a5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:40.562 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,9985865e6944c6a5,78ce420f60cdfb05] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9985865e6944c6a5, duration=1 ms] 2024-10-16 14:05:40.602 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:40.994 INFO 4486 --- [prometheus-minute-schedule-1,7bfd6b0cbf879f45,e407027f342a1c5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:40.995 INFO 4486 --- [prometheus-hour-schedule-1,544ea6fa42e65fc4,c17a18f5834d3a9f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:05:41.000 INFO 4486 --- [prometheus-minute-schedule-1,7bfd6b0cbf879f45,e407027f342a1c5a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:41.431 INFO 4486 --- [ocp-schedules-18,e0829f43397caab4,83cca21e6ef6fe36] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:05:41.436 INFO 4486 --- [ocp-async-2,65b4ea79fb453442,dcbd0697b3ed4e9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:42.093 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,92493cd05887339c,f50968c2b2a51482] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=92493cd05887339c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:42.093 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,92493cd05887339c,f50968c2b2a51482] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=92493cd05887339c, duration=1 ms] 2024-10-16 14:05:42.199 INFO 4486 --- [ocp-schedules-29,1a065ab499c22c84,f6b183641a2f59fa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:05:42.403 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0b27256a1a10d79a,4d32170904579b8e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0b27256a1a10d79a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:42.404 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,0b27256a1a10d79a,4d32170904579b8e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0b27256a1a10d79a, duration=1 ms] 2024-10-16 14:05:43.034 INFO 4486 --- [ocp-async-5,672d32235dce925d,024a2b96303608ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:05:43.436 INFO 4486 --- [ocp-async-3,748103a325c8913c,527a106fb5b13908] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:43.602 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:43.994 INFO 4486 --- [prometheus-minute-schedule-1,e1e2f4c94bbb90d3,ff3d172cd513ac7d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:44.000 INFO 4486 --- [prometheus-minute-schedule-1,e1e2f4c94bbb90d3,ff3d172cd513ac7d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:45.423 INFO 4486 --- [ocp-schedules-16,c1b2aec2f55f096c,9cf6472ae45d2ea8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:05:45.423 INFO 4486 --- [ocp-schedules-16,c31f80fb3417388a,cf49448fe5cb7a03] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:05:45.426 INFO 4486 --- [ocp-schedules-16,c31f80fb3417388a,cf49448fe5cb7a03] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:45.436 INFO 4486 --- [ocp-async-4,726c5c235f73ecd0,9072eca183850865] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:45.454 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.454 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:45.463 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.464 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:45.484 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:45.489 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.489 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.489 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:05:45.490 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:45.517 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:45.517 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.518 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:45.519 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:05:45.522 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.525 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:05:45.526 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:45.540 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.540 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:45.557 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.566 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:45.567 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.568 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:45.568 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:05:45.573 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:05:45.577 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:45.579 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:45.590 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:45.591 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.592 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:05:45.600 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:05:45.620 INFO 4486 --- [ocp-schedules-28,82bd8867a6c92b35,1b09582880f8d120] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:05:45.624 INFO 4486 --- [ocp-schedules-28,82bd8867a6c92b35,1b09582880f8d120] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:05:45.624 INFO 4486 --- [ocp-schedules-28,82bd8867a6c92b35,1b09582880f8d120] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:05:45.628 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:05:45.628 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:05:45.644 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:45.645 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.646 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:05:45.650 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.654 INFO 4486 --- [ocp-schedules-13,443518eee534e470,f6234081700d94f0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:05:45.654 INFO 4486 --- [ocp-schedules-13,443518eee534e470,f6234081700d94f0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:05:45.657 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:05:45.660 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:45.672 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:05:45.682 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:45.684 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.684 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:45.687 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.687 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:05:45.697 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:05:45.697 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:05:45.709 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.709 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:45.733 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.733 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:45.758 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.758 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:05:45.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.784 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:05:45.808 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:05:45.835 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:45.836 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:45.837 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:05:45.842 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:05:45.847 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:05:45.852 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:05:45.854 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:05:45.879 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:05:46.034 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:05:46.061 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:05:46.062 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:46.064 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:05:46.093 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:05:46.093 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:05:46.501 INFO 4486 --- [ocp-schedules-5,d7375f57322cfbea,bd00c8aa85dfe914] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:05:46.533 WARN 4486 --- [scanner-13330613-1,d7375f57322cfbea,c02121f328f9af64] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:05:46.605 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:46.646 INFO 4486 --- [alarm-detect-11,d7375f57322cfbea,d1e69597ffccae87] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058715 ~ 1729058736 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058730, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058730, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058730=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:05:46.994 INFO 4486 --- [prometheus-minute-schedule-1,5d605f07701005b9,a8e02ea347c384a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:47.000 INFO 4486 --- [prometheus-minute-schedule-1,5d605f07701005b9,a8e02ea347c384a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:47.070 INFO 4486 --- [ocp-schedules-27,140518d0e95f77a2,d0278dc7e36b064c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:05:47.177 INFO 4486 --- [ocp-schedules-25,783c21156bb7e916,e530e66ffc08cc7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:05:47.307 INFO 4486 --- [ocp-schedules-9,cac289730bcc2514,97065352bd8b57c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:05:47.436 INFO 4486 --- [ocp-async-7,729f0b87fdf67298,df87c6a0ecb534ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:48.551 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2c05d5e83de2235f,c2f9d929f93faf24] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2c05d5e83de2235f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:48.552 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,2c05d5e83de2235f,c2f9d929f93faf24] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2c05d5e83de2235f, duration=3 ms] 2024-10-16 14:05:48.786 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:48.787 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:05:48.788 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:05:48.789 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:05:48.790 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 14:05:49.074 INFO 4486 --- [ocp-schedules-1,567ae39be10149c5,f3069544e0612646] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 14:05:49.075 INFO 4486 --- [ocp-schedules-1,567ae39be10149c5,f3069544e0612646] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:49.436 INFO 4486 --- [ocp-async-8,9ac51dbea257469c,63aad4702d66887d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:49.606 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:49.994 INFO 4486 --- [prometheus-minute-schedule-1,5ea033174a287914,022a99d8c73029cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:50.002 INFO 4486 --- [prometheus-minute-schedule-1,5ea033174a287914,022a99d8c73029cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:50.567 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3f1e33a44e1e3fc3,53fcc043fa4080aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3f1e33a44e1e3fc3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:50.567 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3f1e33a44e1e3fc3,53fcc043fa4080aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3f1e33a44e1e3fc3, duration=1 ms] 2024-10-16 14:05:50.589 INFO 4486 --- [ocp-schedules-23,5a7823b8f92bb24e,f3ee13e40ac74dc9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:05:50.589 INFO 4486 --- [ocp-schedules-23,5a7823b8f92bb24e,f3ee13e40ac74dc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:05:50.595 INFO 4486 --- [ocp-schedules-23,5a7823b8f92bb24e,f3ee13e40ac74dc9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:05:50.595 INFO 4486 --- [ocp-schedules-23,5a7823b8f92bb24e,f3ee13e40ac74dc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:05:50.599 INFO 4486 --- [ocp-schedules-23,5a7823b8f92bb24e,f3ee13e40ac74dc9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:05:50.599 INFO 4486 --- [ocp-schedules-23,5a7823b8f92bb24e,f3ee13e40ac74dc9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 14:05:51.431 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,96181480f7b94981,1e8962a144de42da] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=96181480f7b94981, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:05:51.431 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,96181480f7b94981,1e8962a144de42da] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=96181480f7b94981, duration=1 ms] 2024-10-16 14:05:51.436 INFO 4486 --- [ocp-schedules-5,1dfc9f5f7a278a38,0e1ad63f4d8221ca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:05:51.437 INFO 4486 --- [ocp-async-1,432e199fe214b1e8,ba3ef7472ee9c16c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:51.738 INFO 4486 --- [ocp-schedules-12,6a93c9cf490122b6,a0211bbccf1bfeb1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:05:51.739 INFO 4486 --- [ocp-schedules-12,6a93c9cf490122b6,a0211bbccf1bfeb1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:05:51.739 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:05:51.739 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:05:51.739 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:05:51.739 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:05:51.739 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:05:51.745 INFO 4486 --- [ocp-schedules-12,6a93c9cf490122b6,a0211bbccf1bfeb1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:05:51.745 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:05:51.745 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:05:51.745 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:05:51.746 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:05:51.746 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:05:51.747 INFO 4486 --- [ocp-schedules-12,6a93c9cf490122b6,a0211bbccf1bfeb1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:05:51.747 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:05:51.747 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:05:51.747 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:05:51.747 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:05:51.748 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:05:51.771 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:05:51.771 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:05:51.771 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:05:51.775 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:05:51.778 INFO 4486 --- [ocp-schedules-12,6a93c9cf490122b6,a0211bbccf1bfeb1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:05:51.781 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:05:51.785 INFO 4486 --- [ocp-schedules-12,6a93c9cf490122b6,a0211bbccf1bfeb1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:05:52.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,48913e936549cbf4,bce50da97c9f0fb7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=48913e936549cbf4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:52.106 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,48913e936549cbf4,bce50da97c9f0fb7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=48913e936549cbf4, duration=1 ms] 2024-10-16 14:05:52.408 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,90466bb28ebd7ba0,ef172a64d1ed3c2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=90466bb28ebd7ba0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:52.409 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,90466bb28ebd7ba0,ef172a64d1ed3c2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=90466bb28ebd7ba0, duration=2 ms] 2024-10-16 14:05:52.608 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:52.994 INFO 4486 --- [prometheus-minute-schedule-1,113c1bc301480051,5fd49650ac544043] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:53.000 INFO 4486 --- [prometheus-minute-schedule-1,113c1bc301480051,5fd49650ac544043] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:53.034 INFO 4486 --- [ocp-async-3,01ec7c0fd1ee37a6,f0c2f617c75bfc97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:05:53.136 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dbcb9e86a160d36a,c3d6bb9e1dc764de] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=dbcb9e86a160d36a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:05:53.136 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,dbcb9e86a160d36a,c3d6bb9e1dc764de] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=dbcb9e86a160d36a, duration=1 ms] 2024-10-16 14:05:53.437 INFO 4486 --- [ocp-async-7,01b516df5ce684f9,54bb755a76bb0b9b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:53.591 INFO 4486 --- [ocp-schedules-17,ce6a1d32696dac0a,01ce5a84cd42e985] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:05:53.593 INFO 4486 --- [ocp-schedules-17,ce6a1d32696dac0a,01ce5a84cd42e985] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:05:53.594 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:05:53.594 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:05:53.594 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:05:55.437 INFO 4486 --- [ocp-async-8,0fac1b60d43ef12f,2dc6d5b7aa254684] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:55.608 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:55.994 INFO 4486 --- [prometheus-minute-schedule-1,c47c18dc81b93c1f,a11272459b05c421] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:56.001 INFO 4486 --- [prometheus-minute-schedule-1,c47c18dc81b93c1f,a11272459b05c421] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:56.738 INFO 4486 --- [ocp-schedules-10,7f6b72134bf51283,bc823285e6ba0d0a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:05:56.760 WARN 4486 --- [scanner-13330921-1,7f6b72134bf51283,117395014f5ad022] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:05:56.826 INFO 4486 --- [alarm-detect-4,7f6b72134bf51283,18b1c40a0646281d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058730 ~ 1729058746 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058745, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058745, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058745=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:05:57.438 INFO 4486 --- [ocp-async-4,fcfaf3f47a0fefcf,3ebad7d6d9988a49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:58.558 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,801b2f40021f0c1f,fd707e1ed4abc403] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=801b2f40021f0c1f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:05:58.558 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,801b2f40021f0c1f,fd707e1ed4abc403] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=801b2f40021f0c1f, duration=1 ms] 2024-10-16 14:05:58.609 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:05:58.994 INFO 4486 --- [prometheus-minute-schedule-1,30c2bf9d812de7d5,2aee8e199d0165c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058760, interval=60 2024-10-16 14:05:59.001 INFO 4486 --- [prometheus-minute-schedule-1,30c2bf9d812de7d5,2aee8e199d0165c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:05:59.106 INFO 4486 --- [ocp-schedules-1,567ae39be10149c5,f3069544e0612646] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 14:05:59.621 INFO 4486 --- [ocp-async-10,2c3c0f4f537a1471,faaed0b2c54316bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:05:59.624 INFO 4486 --- [ocp-schedules-8,7013e69d166ae339,2219828cc74128f2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:06:00.572 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,7d670fe0e9278165,bdde02024a30cd76] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7d670fe0e9278165, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:00.573 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,7d670fe0e9278165,bdde02024a30cd76] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7d670fe0e9278165, duration=3 ms] 2024-10-16 14:06:01.442 INFO 4486 --- [ocp-schedules-27,6ec3c0dd832807c3,a5bfcb1acda67f6b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:06:01.610 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:01.621 INFO 4486 --- [ocp-async-6,c4f45bd8313e5449,bad9f6408f9bc9a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:01.994 INFO 4486 --- [prometheus-minute-schedule-1,71e2a77f841fe0df,75eb0f59fffb67a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:02.000 INFO 4486 --- [prometheus-minute-schedule-1,71e2a77f841fe0df,75eb0f59fffb67a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:02.114 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,46c394f16c4933ea,fc5899220b0b9309] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=46c394f16c4933ea, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:02.114 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,46c394f16c4933ea,fc5899220b0b9309] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=46c394f16c4933ea, duration=1 ms] 2024-10-16 14:06:02.413 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,304471145c74ab09,d535a3e53fd33b73] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=304471145c74ab09, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:02.413 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,304471145c74ab09,d535a3e53fd33b73] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=304471145c74ab09, duration=1 ms] 2024-10-16 14:06:02.953 INFO 4486 --- [ocp-schedules-18,44430225a4ccffe5,e42d41d7b0b0ad62] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:06:02.953+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:06:02.958 INFO 4486 --- [ocp-schedules-18,44430225a4ccffe5,e42d41d7b0b0ad62] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:06:02.962 INFO 4486 --- [ocp-schedules-18,44430225a4ccffe5,e42d41d7b0b0ad62] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster obce_3zones for slowSqlStat, sql count is 0. 2024-10-16 14:06:02.974 INFO 4486 --- [ocp-schedules-18,44430225a4ccffe5,e42d41d7b0b0ad62] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:06:03.034 INFO 4486 --- [ocp-async-2,de2c99c8a595619e,537cf434f1e2c0b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:06:03.594 INFO 4486 --- [ocp-schedules-17,ce6a1d32696dac0a,01ce5a84cd42e985] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:06:03.621 INFO 4486 --- [ocp-async-3,3b9051fab5b5c07b,14178c1d3105946c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:04.611 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:04.994 INFO 4486 --- [prometheus-minute-schedule-1,5adc4949413a2ad8,8d5a184f3bc9b18a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:05.000 INFO 4486 --- [prometheus-minute-schedule-1,5adc4949413a2ad8,8d5a184f3bc9b18a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:05.621 INFO 4486 --- [ocp-async-2,3c2c1901d03464ef,e24cc83e4ba10040] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:06.895 INFO 4486 --- [ocp-schedules-2,c994ff80b3007881,a84c1a4c06018b14] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:06:06.954 WARN 4486 --- [scanner-13331190-1,c994ff80b3007881,e963b63837e6cd3a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:06:07.071 INFO 4486 --- [ocp-schedules-13,1f1e2051448121c0,b29d91faa75d4f4b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:06:07.071 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:06:07.075 INFO 4486 --- [ocp-schedules-13,1f1e2051448121c0,b29d91faa75d4f4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:07.083 INFO 4486 --- [ocp-schedules-24,6483d225e06fdf40,1dc5174f5f583d1c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:06:07.084 INFO 4486 --- [ocp-schedules-24,6483d225e06fdf40,1dc5174f5f583d1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:07.098 INFO 4486 --- [ocp-schedules-27,17363b5574015e25,543d6a39306d5eeb] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:06:07.097 INFO 4486 --- [ocp-schedules-15,ae90f59a2d3a0082,5daf536ecde4ef60] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:06:07.098 INFO 4486 --- [ocp-schedules-23,47de2e36aac528f2,09306c577e7aa4b8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:06:07.099 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:06:07.099 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:06:07.107 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:06:07.111 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:07.112 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:06:07.118 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:06:07.118 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:07.120 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:06:07.120 INFO 4486 --- [host-check-agent-status1,ae90f59a2d3a0082,8eb497246ec03b25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:06:07.121 INFO 4486 --- [host-check-agent-status2,ae90f59a2d3a0082,e2cccf4feeac3759] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:06:07.122 INFO 4486 --- [host-check-agent-status4,ae90f59a2d3a0082,af04dc65cf8431e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:06:07.122 INFO 4486 --- [host-check-agent-status3,ae90f59a2d3a0082,ad00f1747b3ae5d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:06:07.124 INFO 4486 --- [host-check-clockdiff1,17363b5574015e25,caeb2d375729a9f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:06:07.124 INFO 4486 --- [host-check-clockdiff3,17363b5574015e25,098970bd7e49ad86] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:06:07.124 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:06:07.125 INFO 4486 --- [host-check-clockdiff2,17363b5574015e25,c941722292f29368] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:06:07.128 INFO 4486 --- [host-check-agent-status1,ae90f59a2d3a0082,8eb497246ec03b25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:07.128 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:06:07.128 INFO 4486 --- [host-check-agent-status2,ae90f59a2d3a0082,e2cccf4feeac3759] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:06:07.130 INFO 4486 --- [host-check-agent-status2,ae90f59a2d3a0082,e2cccf4feeac3759] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:06:07.130 INFO 4486 --- [host-check-agent-status1,ae90f59a2d3a0082,8eb497246ec03b25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:07.130 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:06:07.131 INFO 4486 --- [host-check-agent-status3,ae90f59a2d3a0082,ad00f1747b3ae5d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:06:07.132 INFO 4486 --- [host-check-agent-status4,ae90f59a2d3a0082,af04dc65cf8431e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:06:07.134 INFO 4486 --- [host-check-agent-status3,ae90f59a2d3a0082,ad00f1747b3ae5d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:06:07.135 INFO 4486 --- [host-check-agent-status4,ae90f59a2d3a0082,af04dc65cf8431e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:06:07.138 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.138 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:07.160 INFO 4486 --- [host-check-clockdiff4,17363b5574015e25,42d1da7cc60ed034] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:06:07.208 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:06:07.219 INFO 4486 --- [host-check-clockdiff3,17363b5574015e25,098970bd7e49ad86] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058767 4 4 , error= 2024-10-16 14:06:07.221 INFO 4486 --- [host-check-clockdiff1,17363b5574015e25,caeb2d375729a9f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058767 0 0 , error= 2024-10-16 14:06:07.224 INFO 4486 --- [host-check-clockdiff2,17363b5574015e25,c941722292f29368] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058767 4 4 , error= 2024-10-16 14:06:07.232 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:06:07.234 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:06:07.238 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.239 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:07.240 INFO 4486 --- [host-check-agent-status2,ae90f59a2d3a0082,e2cccf4feeac3759] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:06:07.241 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:06:07.242 INFO 4486 --- [host-check-agent-status3,ae90f59a2d3a0082,ad00f1747b3ae5d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:06:07.243 INFO 4486 --- [host-check-agent-status4,ae90f59a2d3a0082,af04dc65cf8431e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:06:07.243 INFO 4486 --- [host-check-agent-status1,ae90f59a2d3a0082,8eb497246ec03b25] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:06:07.244 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:06:07.247889452+08:00 2024-10-16 14:06:07.244 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:06:07.245 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:06:07.244+08:00 2024-10-16 14:06:07.245 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-agent-status2,ae90f59a2d3a0082,e2cccf4feeac3759] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-agent-status2,ae90f59a2d3a0082,e2cccf4feeac3759] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-agent-status2,ae90f59a2d3a0082,e2cccf4feeac3759] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-agent-status3,ae90f59a2d3a0082,ad00f1747b3ae5d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-agent-status3,ae90f59a2d3a0082,ad00f1747b3ae5d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-agent-status3,ae90f59a2d3a0082,ad00f1747b3ae5d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:06:07.251571490+08:00 2024-10-16 14:06:07.247 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:06:07.244+08:00 and status AVAILABLE 2024-10-16 14:06:07.248 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:06:07.247+08:00 2024-10-16 14:06:07.249 INFO 4486 --- [host-check-agent-status4,ae90f59a2d3a0082,af04dc65cf8431e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:06:07.249 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.249 INFO 4486 --- [host-check-agent-status4,ae90f59a2d3a0082,af04dc65cf8431e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:06:07.249 INFO 4486 --- [host-check-agent-status4,ae90f59a2d3a0082,af04dc65cf8431e1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:06:07.250 INFO 4486 --- [host-check-agent-status1,ae90f59a2d3a0082,8eb497246ec03b25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:06:07.250 INFO 4486 --- [host-check-agent-status1,ae90f59a2d3a0082,8eb497246ec03b25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:06:07.250 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:06:07.250 INFO 4486 --- [host-check-agent-status1,ae90f59a2d3a0082,8eb497246ec03b25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:06:07.250 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:06:07.251 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:06:07.250682900+08:00 2024-10-16 14:06:07.251 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:06:07.247+08:00 and status AVAILABLE 2024-10-16 14:06:07.251 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:06:07.251+08:00 2024-10-16 14:06:07.253 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:06:07.254 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:06:07.251+08:00 and status AVAILABLE 2024-10-16 14:06:07.255 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:06:07.257 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:06:07.259 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:06:07.260 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:06:07.260 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:06:07.263451175+08:00 2024-10-16 14:06:07.261 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:06:07.260+08:00 2024-10-16 14:06:07.261 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:06:07.261 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:06:07.262 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:06:07.263 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:06:07.260+08:00 and status AVAILABLE 2024-10-16 14:06:07.266 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:06:07.266 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:07.268 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:07.268 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.268 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:07.276 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:06:07.278 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:06:07.278 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.279 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:07.285 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:07.286 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:07.288 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:07.289 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.290 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:07.293 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.295 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:06:07.302 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:07.307 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:07.308 INFO 4486 --- [ocp-schedules-22,5eb22a60f00f8959,1cabcedbc6c99ae3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:06:07.315 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:06:07.327 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:06:07.327 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:06:07.337 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:06:07.339 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:06:07.342 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:06:07.344 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:07.347 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:06:07.352 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:06:07.356 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:07.364 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:07.366 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:07.373 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:06:07.384 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:06:07.392 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:06:07.399 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:06:07.400 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:07.405 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:06:07.411 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:06:07.419 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:07.427 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:06:07.428 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:06:07.430 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:07.435 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:06:07.438 INFO 4486 --- [host-check-host-status3,47de2e36aac528f2,5bbbccf0da7281ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:06:07.449 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:07.459 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:06:07.462 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:06:07.470 INFO 4486 --- [host-check-host-status2,47de2e36aac528f2,ecc7fdf66e300a2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:06:07.473 INFO 4486 --- [host-check-host-status4,47de2e36aac528f2,9d77c23cd3aa23ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:06:07.500 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:06:07.511 INFO 4486 --- [host-check-host-status1,47de2e36aac528f2,d433ec5b4d4550cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:06:07.612 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:07.622 INFO 4486 --- [ocp-async-7,9ae431fca1dcf651,3e05de192ae2f0fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:07.994 INFO 4486 --- [prometheus-minute-schedule-1,a5fe862a28f8c193,2f2d49add478f989] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:08.000 INFO 4486 --- [prometheus-minute-schedule-1,a5fe862a28f8c193,2f2d49add478f989] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:08.120 INFO 4486 --- [host-check-clockdiff4,17363b5574015e25,42d1da7cc60ed034] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058768 4 4 , error= 2024-10-16 14:06:08.563 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8163f854d53e592f,17052774c69588e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8163f854d53e592f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:08.563 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,8163f854d53e592f,17052774c69588e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8163f854d53e592f, duration=1 ms] 2024-10-16 14:06:09.622 INFO 4486 --- [ocp-async-1,2eb6a002d0feaba6,66002d9ff3f1ef7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:10.006 INFO 4486 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:06:10.007 INFO 4486 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:06:10.024 INFO 4486 --- [Check obproxy1,f34a58dc4dd91b5d,6e55662b5d0b24e7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:06:10.025 INFO 4486 --- [Check obproxy2,af01a908529d57a7,1c63bea77157e31a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:06:10.025 INFO 4486 --- [Check obproxy1,f34a58dc4dd91b5d,6e55662b5d0b24e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:10.025 INFO 4486 --- [Check obproxy2,af01a908529d57a7,1c63bea77157e31a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:10.030 INFO 4486 --- [Collect obproxy2,1c4d4672493e9d4c,6f7fbb8137b3b765] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:06:10.031 INFO 4486 --- [Collect obproxy2,1c4d4672493e9d4c,6f7fbb8137b3b765] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:10.034 INFO 4486 --- [Collect obproxy1,4c90c46f16f9610a,21365f1b41f74f32] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:06:10.034 INFO 4486 --- [Collect obproxy1,4c90c46f16f9610a,21365f1b41f74f32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:10.051 INFO 4486 --- [Collect obproxy2,1c4d4672493e9d4c,6f7fbb8137b3b765] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:06:10.054 INFO 4486 --- [Check obproxy1,f34a58dc4dd91b5d,6e55662b5d0b24e7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:06:10.054 INFO 4486 --- [Check obproxy1,f34a58dc4dd91b5d,6e55662b5d0b24e7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:06:10.054 INFO 4486 --- [Collect obproxy1,4c90c46f16f9610a,21365f1b41f74f32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:06:10.056 INFO 4486 --- [Check obproxy2,af01a908529d57a7,1c63bea77157e31a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:06:10.056 INFO 4486 --- [Check obproxy2,af01a908529d57a7,1c63bea77157e31a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:06:10.057 INFO 4486 --- [Collect obproxy2,1c4d4672493e9d4c,6f7fbb8137b3b765] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:06:10.057 INFO 4486 --- [Collect obproxy1,4c90c46f16f9610a,21365f1b41f74f32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:06:10.060 INFO 4486 --- [Collect obproxy1,4c90c46f16f9610a,21365f1b41f74f32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:06:10.061 INFO 4486 --- [Collect obproxy2,1c4d4672493e9d4c,6f7fbb8137b3b765] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:06:10.577 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,7be4399911794296,38585637c82fabda] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7be4399911794296, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:10.577 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,7be4399911794296,38585637c82fabda] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7be4399911794296, duration=1 ms] 2024-10-16 14:06:10.613 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:10.981 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:06:10.994 INFO 4486 --- [prometheus-minute-schedule-1,6261bbf32e153ddf,7317879f56ce94be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:11.000 INFO 4486 --- [prometheus-minute-schedule-1,6261bbf32e153ddf,7317879f56ce94be] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:06:11.449 INFO 4486 --- [ocp-schedules-7,e12b4bba717391d0,8f713824dda9922c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:06:11.622 INFO 4486 --- [ocp-async-5,ea04c684c29d32c1,da26e2667743652a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:12.118 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,f39ad1421fd2521e,78fd7cba7807346e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f39ad1421fd2521e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:12.118 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,f39ad1421fd2521e,78fd7cba7807346e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f39ad1421fd2521e, duration=1 ms] 2024-10-16 14:06:12.421 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9ff51e039a4e57ed,d38eb2a219e8a133] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9ff51e039a4e57ed, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:12.421 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9ff51e039a4e57ed,d38eb2a219e8a133] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9ff51e039a4e57ed, duration=5 ms] 2024-10-16 14:06:13.034 INFO 4486 --- [ocp-async-6,9540f21f89f47551,397a188eea54b2c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:06:13.614 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:13.622 INFO 4486 --- [ocp-async-2,1d604bbfead05d76,b4b67a7e94405219] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:13.994 INFO 4486 --- [prometheus-minute-schedule-1,9a8978333b84df5a,81620f1c093eeb75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:14.000 INFO 4486 --- [prometheus-minute-schedule-1,9a8978333b84df5a,81620f1c093eeb75] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:14.752 ERROR 4486 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=1694585) Duplicate entry '586679390278586368-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.751170] [YB420AC531B8-00060B7DA1ABF563-0-0]; nested exception is java.sql.BatchUpdateException: (conn=1694585) Duplicate entry '586679390278586368-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.751170] [YB420AC531B8-00060B7DA1ABF563-0-0] 2024-10-16 14:06:14.753 ERROR 4486 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2457465) Duplicate entry '587073990835113984-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.752883] [YB420AC531B8-00060B7DAD2B8B3D-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2457465) Duplicate entry '587073990835113984-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.752883] [YB420AC531B8-00060B7DAD2B8B3D-0-0] 2024-10-16 14:06:14.753 ERROR 4486 --- [metric-archive-second-20,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2949508) Duplicate entry '586679389997568000-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.753177] [YB420AC531B8-00060B7D84EAB4F1-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2949508) Duplicate entry '586679389997568000-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.753177] [YB420AC531B8-00060B7D84EAB4F1-0-0] 2024-10-16 14:06:14.755 ERROR 4486 --- [metric-archive-second-12,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=2734081) Duplicate entry '586679390194700288-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.754431] [YB420AC531B8-00060B7DB03B35BA-0-0]; nested exception is java.sql.BatchUpdateException: (conn=2734081) Duplicate entry '586679390194700288-1729058700' for key 'PRIMARY' [10.197.49.184:2882] [2024-10-16 14:06:14.754431] [YB420AC531B8-00060B7DB03B35BA-0-0] 2024-10-16 14:06:14.782 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058700, nodeCount=1000, affectRows=1000 2024-10-16 14:06:14.783 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058700, nodeCount=1000, affectRows=1000 2024-10-16 14:06:14.786 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058700, nodeCount=999, validCount=999 2024-10-16 14:06:14.787 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058700, nodeCount=1000, validCount=1000 2024-10-16 14:06:14.787 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058700, nodeCount=1000, affectRows=1000 2024-10-16 14:06:14.790 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058700, nodeCount=1000, validCount=1000 2024-10-16 14:06:14.813 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058700, nodeCount=83, affectRows=83 2024-10-16 14:06:14.813 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058700, nodeCount=83, validCount=83 2024-10-16 14:06:14.829 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058700, nodeCount=112, affectRows=112 2024-10-16 14:06:14.829 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058700, nodeCount=111, validCount=111 2024-10-16 14:06:14.840 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058700, nodeCount=121, affectRows=121 2024-10-16 14:06:14.840 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058700, nodeCount=120, validCount=120 2024-10-16 14:06:14.847 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058700, nodeCount=135, affectRows=135 2024-10-16 14:06:14.848 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058700, nodeCount=135, validCount=135 2024-10-16 14:06:15.387 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058700, nodeCount=477, affectRows=477 2024-10-16 14:06:15.623 INFO 4486 --- [ocp-async-6,87df1ecc09b29fd6,2910074edec396b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:16.065 INFO 4486 --- [metric-archive-async-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058700, nodeCount=477, validCount=477 2024-10-16 14:06:16.093 INFO 4486 --- [ocp-schedules-16,c31f80fb3417388a,cf49448fe5cb7a03] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:06:16.094 INFO 4486 --- [ocp-schedules-16,bc42ab9e7edc088f,0805a0222418c472] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:06:16.095 INFO 4486 --- [ocp-schedules-16,bc42ab9e7edc088f,0805a0222418c472] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:16.122 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.123 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:16.133 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.134 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:16.156 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.157 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.157 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:16.158 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.158 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:06:16.188 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.189 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.189 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:16.190 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:06:16.191 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.194 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:06:16.196 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:16.207 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.207 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:16.220 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.229 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.230 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.230 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:16.232 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:06:16.238 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:06:16.243 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:16.245 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:16.267 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.269 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.269 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:06:16.276 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:06:16.297 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:06:16.297 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:06:16.320 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.321 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.322 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:16.325 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.329 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:06:16.332 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:16.356 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:06:16.360 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.360 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:16.366 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.367 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.368 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:06:16.378 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:06:16.378 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:06:16.389 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.389 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:16.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.415 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:16.443 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.444 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:16.469 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.469 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:16.494 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:16.522 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.523 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.524 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:16.529 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:06:16.534 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:06:16.539 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:06:16.541 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:06:16.563 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:06:16.615 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:16.694 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:06:16.719 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:16.720 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:16.721 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:06:16.749 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:06:16.749 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:06:16.994 INFO 4486 --- [prometheus-minute-schedule-1,25933543c3865682,bc02c356aaba5c1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:16.999 INFO 4486 --- [prometheus-minute-schedule-1,25933543c3865682,bc02c356aaba5c1a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:17.120 INFO 4486 --- [ocp-schedules-13,1f1e2051448121c0,b29d91faa75d4f4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:06:17.239 INFO 4486 --- [ocp-schedules-23,9b2a43fb45c134c1,22b8bf1a14ec2f1f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:06:17.255 WARN 4486 --- [scanner-13332324-1,9b2a43fb45c134c1,d62a641f6af9d13e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:06:17.259 INFO 4486 --- [ocp-schedules-24,6483d225e06fdf40,1dc5174f5f583d1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:06:17.313 INFO 4486 --- [ocp-schedules-22,ffbcd7bf45d5ff95,da2ddf05d8d90e6f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:06:17.315 INFO 4486 --- [alarm-detect-19,9b2a43fb45c134c1,a9d51a496fa5d6f1] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058745 ~ 1729058767 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058760, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058760, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058760=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:06:17.623 INFO 4486 --- [ocp-async-6,2845e99e957d4519,a56e289c07bc6f80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:18.567 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6d9e306dec04586d,31cbb8cb9c6778b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6d9e306dec04586d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:18.568 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,6d9e306dec04586d,31cbb8cb9c6778b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6d9e306dec04586d, duration=2 ms] 2024-10-16 14:06:19.616 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:19.623 INFO 4486 --- [ocp-async-7,1c6c7d5b459bc339,dd7c161740304cae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:19.630 INFO 4486 --- [ocp-schedules-19,d8b53b663db25f8d,74768f922f3fe123] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:06:19.994 INFO 4486 --- [prometheus-minute-schedule-1,26bd8a0e295248c2,36cb67375e9b1095] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:20.000 INFO 4486 --- [prometheus-minute-schedule-1,26bd8a0e295248c2,36cb67375e9b1095] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:20.582 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,31923e803dba2235,a0760179032322d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=31923e803dba2235, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:20.582 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,31923e803dba2235,a0760179032322d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=31923e803dba2235, duration=1 ms] 2024-10-16 14:06:20.814 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 14:06:20.996 INFO 4486 --- [prometheus-cache-monitor1,d2b07d04ae6541d8,ac3c2949082f1de9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13800,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8131} 2024-10-16 14:06:20.997 INFO 4486 --- [prometheus-cache-monitor1,d2b07d04ae6541d8,ac3c2949082f1de9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:06:21.464 INFO 4486 --- [ocp-schedules-30,746f04561efb5af4,ed109f602e95074a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:06:21.623 INFO 4486 --- [ocp-async-8,a0fd41f136ca3776,3d89a39c617fdadd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:22.123 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3c8152713236c6bf,b2afaee0d44100e7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3c8152713236c6bf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:22.123 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,3c8152713236c6bf,b2afaee0d44100e7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3c8152713236c6bf, duration=1 ms] 2024-10-16 14:06:22.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b8393ba4d43309c5,14543794f0db8d55] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b8393ba4d43309c5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:22.427 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b8393ba4d43309c5,14543794f0db8d55] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b8393ba4d43309c5, duration=1 ms] 2024-10-16 14:06:22.617 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:22.994 INFO 4486 --- [prometheus-minute-schedule-1,3f72df7dc7f2ef04,a8b939437aa67230] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:23.000 INFO 4486 --- [prometheus-minute-schedule-1,3f72df7dc7f2ef04,a8b939437aa67230] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:23.034 INFO 4486 --- [ocp-async-10,94f408decd1e7b92,6ac023e999a843e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:06:23.623 INFO 4486 --- [ocp-async-3,0e102dc6bdfd6a72,37ac983d03c8f676] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:24.163 INFO 4486 --- [ocp-schedules-27,74da86b33216347d,8dcd6b90cf14706a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:06:25.385 INFO 4486 --- [ocp-schedules-18,0a0562c9cb9b0d31,d4c69e2cd62713ef] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:06:25.618 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:25.624 INFO 4486 --- [ocp-async-5,380f6d54edcf19f0,d8262dd7cff9d7b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:25.994 INFO 4486 --- [prometheus-minute-schedule-1,4dcb889404ca387d,2d8945081b8d81d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:26.000 INFO 4486 --- [prometheus-minute-schedule-1,4dcb889404ca387d,2d8945081b8d81d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:27.142 INFO 4486 --- [ocp-schedules-5,22630eca215490a8,413b9be8cba95388] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:06:27.142 INFO 4486 --- [ocp-schedules-5,22630eca215490a8,413b9be8cba95388] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:06:27.174 INFO 4486 --- [Collect all tenants major compactions2,22630eca215490a8,cea234cc1fb2dbdc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:06:27.174 INFO 4486 --- [Collect all tenants major compactions1,22630eca215490a8,b82b7ee84296c1d7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:06:27.174 INFO 4486 --- [Collect all tenants major compactions5,22630eca215490a8,384b5dfa8fbe2d4b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:06:27.174 INFO 4486 --- [Collect all tenants major compactions4,22630eca215490a8,0498f5b0a8d32e1c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:06:27.174 INFO 4486 --- [Collect all tenants major compactions3,22630eca215490a8,bf2b75074eb2b5bf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:06:27.184 INFO 4486 --- [Collect all tenants major compactions1,22630eca215490a8,b82b7ee84296c1d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:27.185 INFO 4486 --- [Collect all tenants major compactions2,22630eca215490a8,cea234cc1fb2dbdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:27.186 INFO 4486 --- [Collect all tenants major compactions2,22630eca215490a8,cea234cc1fb2dbdc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:06:27.186 INFO 4486 --- [Collect all tenants major compactions1,22630eca215490a8,b82b7ee84296c1d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:06:27.189 INFO 4486 --- [Collect all tenants major compactions1,22630eca215490a8,b82b7ee84296c1d7] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:06:27.190 INFO 4486 --- [Collect all tenants major compactions2,22630eca215490a8,cea234cc1fb2dbdc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:06:27.190 INFO 4486 --- [Collect all tenants major compactions1,22630eca215490a8,b82b7ee84296c1d7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:06:27.190 INFO 4486 --- [Collect all tenants major compactions2,22630eca215490a8,cea234cc1fb2dbdc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:06:27.193 INFO 4486 --- [Collect all tenants major compactions3,22630eca215490a8,bf2b75074eb2b5bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:27.195 INFO 4486 --- [Collect all tenants major compactions3,22630eca215490a8,bf2b75074eb2b5bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:06:27.199 INFO 4486 --- [Collect all tenants major compactions3,22630eca215490a8,bf2b75074eb2b5bf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:06:27.199 INFO 4486 --- [Collect all tenants major compactions3,22630eca215490a8,bf2b75074eb2b5bf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:06:27.203 INFO 4486 --- [Collect all tenants major compactions5,22630eca215490a8,384b5dfa8fbe2d4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:27.204 INFO 4486 --- [Collect all tenants major compactions4,22630eca215490a8,0498f5b0a8d32e1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:27.205 INFO 4486 --- [Collect all tenants major compactions5,22630eca215490a8,384b5dfa8fbe2d4b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:06:27.206 INFO 4486 --- [Collect all tenants major compactions4,22630eca215490a8,0498f5b0a8d32e1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:06:27.209 INFO 4486 --- [Collect all tenants major compactions5,22630eca215490a8,384b5dfa8fbe2d4b] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:06:27.210 INFO 4486 --- [Collect all tenants major compactions5,22630eca215490a8,384b5dfa8fbe2d4b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:06:27.210 INFO 4486 --- [Collect all tenants major compactions4,22630eca215490a8,0498f5b0a8d32e1c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:06:27.210 INFO 4486 --- [Collect all tenants major compactions4,22630eca215490a8,0498f5b0a8d32e1c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:06:27.211 INFO 4486 --- [ocp-schedules-5,22630eca215490a8,413b9be8cba95388] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:06:27.365 INFO 4486 --- [ocp-schedules-15,d16b9520d1a7ee73,fa34c894985be495] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:06:27.394 WARN 4486 --- [scanner-13332552-1,d16b9520d1a7ee73,b2ddbc8a7c9c6ab0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:06:27.470 INFO 4486 --- [alarm-detect-5,d16b9520d1a7ee73,719b607b9f677fc7] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058760 ~ 1729058777 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058775, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058775, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058775=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:06:27.624 INFO 4486 --- [ocp-async-2,17db2f963ad9ebb6,d65a5bcca61ca2a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:28.042 INFO 4486 --- [ocp-schedules-8,f0d2f2ef98db3b6a,1f58fbcf4a8fa96a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:06:28.042+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:06:28.045 INFO 4486 --- [ocp-schedules-8,f0d2f2ef98db3b6a,1f58fbcf4a8fa96a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:06:28.045 INFO 4486 --- [ocp-schedules-8,f0d2f2ef98db3b6a,1f58fbcf4a8fa96a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:06:28.046 INFO 4486 --- [ocp-schedules-8,f0d2f2ef98db3b6a,1f58fbcf4a8fa96a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:06:28.574 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9c0c23833249608e,3a4f12de9d69b440] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=9c0c23833249608e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:28.574 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,9c0c23833249608e,3a4f12de9d69b440] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=9c0c23833249608e, duration=3 ms] 2024-10-16 14:06:28.620 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:28.994 INFO 4486 --- [prometheus-minute-schedule-1,d072219f30781cb1,b90046d347a36245] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:28.999 INFO 4486 --- [prometheus-minute-schedule-1,d072219f30781cb1,b90046d347a36245] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:29.624 INFO 4486 --- [ocp-async-6,1fb1436671c364ff,e4dec25e5ebe7598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:29.910 INFO 4486 --- [ocp-schedules-29,179d8a7153b3ff0a,af9e6975afcfd491] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 14:06:29.911 INFO 4486 --- [ocp-schedules-29,179d8a7153b3ff0a,af9e6975afcfd491] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:29.948 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:06:29.949 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:29.953 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:06:29.962 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:06:29.963 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:29.964 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:29.964 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:06:29.967 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:06:29.973 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:29.973 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:06:29.981 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:29.981 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:06:29.987 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:29.988 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:06:29.991 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:29.991 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:06:30.001 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.001 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.001 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:30.002 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:06:30.003 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.003 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:06:30.007 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:06:30.009 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.010 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:30.013 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.014 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:30.017 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.018 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:06:30.019 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.020 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:06:30.025 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.026 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:30.027 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.028 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:06:30.033 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.033 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:06:30.036 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.037 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:06:30.044 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.045 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:06:30.049 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.050 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:06:30.052 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.052 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.053 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.054 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.054 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:06:30.055 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.055 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.056 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.057 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.057 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.058 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.059 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.060 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.060 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.061 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.061 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.062 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.062 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:06:30.063 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.064 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.064 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.065 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.066 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.066 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.067 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.068 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.068 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.069 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.070 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 14:06:30.074 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.075 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:06:30.078 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.078 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:06:30.079 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:30.080 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.082 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.085 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.087 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.089 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.090 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.090 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:30.093 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.095 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:30.095 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.095 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.096 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:06:30.096 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.099 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.100 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.100 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:30.101 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.103 INFO 4486 --- [prometheus-second-schedule-1,bb75b460ff4c137d,9055c442dbf6fbfb] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:06:30.105 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.106 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.107 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 14:06:30.109 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.110 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.114 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 14:06:30.114 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:06:30.114 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.115 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.118 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.121 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.125 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 14:06:30.128 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:06:30.134 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.135 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.138 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.140 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.143 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.145 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.147 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.148 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.152 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.153 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.156 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:30.156 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.157 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.157 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.158 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:06:30.160 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.161 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:06:30.164 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.165 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:06:30.168 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.169 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:06:30.171 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.171 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 14:06:30.171 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.171 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 14:06:30.171 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.171 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 14:06:30.172 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 14:06:30.174 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:06:30.175 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 14:06:30.175 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 14:06:30.229 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:30.231 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:30.232 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:06:30.240 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.241 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 14:06:30.241 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.241 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 14:06:30.241 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:06:30.241 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 14:06:30.244 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 14:06:30.244 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:06:30.593 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3c4aa1eca294c9c5,b7055b465e16b627] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3c4aa1eca294c9c5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:30.593 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,3c4aa1eca294c9c5,b7055b465e16b627] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3c4aa1eca294c9c5, duration=2 ms] 2024-10-16 14:06:31.470 INFO 4486 --- [ocp-schedules-17,51ce56a2128218b5,65bc9390dd545064] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:06:31.620 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:31.625 INFO 4486 --- [ocp-async-10,e4f9c88ed8710af3,7fe3021bf2b1a0b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:31.994 INFO 4486 --- [prometheus-minute-schedule-1,625a32bf6d092524,d585c6f2174afb8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:32.001 INFO 4486 --- [prometheus-minute-schedule-1,625a32bf6d092524,d585c6f2174afb8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:32.156 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3f04f6f793321a2b,43ed59c8ab6694ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3f04f6f793321a2b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:32.157 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,3f04f6f793321a2b,43ed59c8ab6694ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3f04f6f793321a2b, duration=31 ms] 2024-10-16 14:06:32.434 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0089a2f91436838b,92ad7fa9572b49b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0089a2f91436838b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:32.434 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,0089a2f91436838b,92ad7fa9572b49b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0089a2f91436838b, duration=1 ms] 2024-10-16 14:06:33.035 INFO 4486 --- [ocp-async-1,665ede5a07db9184,381b04822e26dbea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:06:33.625 INFO 4486 --- [ocp-async-9,f31216f9281d2be9,262ff51a32589bfa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:33.797 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2024-10-16 14:06:33.800 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=observer, size=5 2024-10-16 14:06:33.801 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_count|app=HOST|name=obproxy, size=3 2024-10-16 14:06:33.802 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=observer, size=5 2024-10-16 14:06:33.804 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_cpu_percent|app=HOST|name=obproxy, size=3 2024-10-16 14:06:33.805 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=observer, size=5 2024-10-16 14:06:33.806 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_exists|app=HOST|name=obproxy, size=3 2024-10-16 14:06:33.808 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=observer, size=5 2024-10-16 14:06:33.809 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_fd_count|app=HOST|name=obproxy, size=3 2024-10-16 14:06:33.810 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=observer, size=5 2024-10-16 14:06:33.811 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_memory_rss_bytes|app=HOST|name=obproxy, size=3 2024-10-16 14:06:33.813 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=observer, size=5 2024-10-16 14:06:33.814 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_thread_count|app=HOST|name=obproxy, size=3 2024-10-16 14:06:33.815 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=observer, size=5 2024-10-16 14:06:33.816 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Delete deletable seriesKey, metricName=process_uptime_seconds|app=HOST|name=obproxy, size=3 2024-10-16 14:06:33.817 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2024-10-16 14:06:33.817 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=40024 2024-10-16 14:06:33.818 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,736f32187d760ca3,ab626e29f356419a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=736f32187d760ca3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:06:33.819 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,736f32187d760ca3,ab626e29f356419a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=736f32187d760ca3, duration=1 ms] 2024-10-16 14:06:34.523 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,7e5b5f048c32617c,30be59b5e45b25df] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=7e5b5f048c32617c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:06:34.524 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,7e5b5f048c32617c,30be59b5e45b25df] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=7e5b5f048c32617c, duration=2 ms] 2024-10-16 14:06:34.621 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:34.994 INFO 4486 --- [prometheus-minute-schedule-1,b4f6cd2325ebfd80,a801eafcc9167aec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:35.000 INFO 4486 --- [prometheus-minute-schedule-1,b4f6cd2325ebfd80,a801eafcc9167aec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703043729918332928 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703043729918332929 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703043729918332930 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703043729914138628 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703043729914138624 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703043729914138625 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703043729914138626 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703043729914138627 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703043729926721536 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703043729926721537 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703043729922527232 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703043729922527233 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703043729901555712 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703043729901555713 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703043729901555714 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703043729897361408 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703043729897361409 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703043729897361410 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703043729909944320 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703043729909944321 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703043729905750020 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703043729905750016 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703043729905750017 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703043729905750018 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703043729905750019 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703043729884778500 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703043729884778496 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703043729884778497 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703043729884778498 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703043729884778499 2024-10-16 14:06:35.098 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703043729880584192 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703043729880584193 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703043729893167108 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.173, id=703043729893167109 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone1|svr_ip=10.197.49.171, id=703043729893167104 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.152, id=703043729893167105 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703043729893167106 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703043729893167107 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703043729868001280 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=|obproxy_cluster_id=0|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.172, id=703043729868001281 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.153, id=703043729876389888 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703043729872195584 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703043729872195585 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703043729851224064 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone3|svr_ip=10.197.49.178, id=703043729859612672 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703043729859612673 2024-10-16 14:06:35.099 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703043729834446848 2024-10-16 14:06:35.157 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703043729842835456 2024-10-16 14:06:35.157 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=observer|ob_cluster_id=1701313639|ob_cluster_name=obce_3zones|obproxy_cluster=obce_3zones|obproxy_cluster_id=2|obregion=obce_3zones|obzone=zone2|svr_ip=10.197.49.177, id=703043729826058240 2024-10-16 14:06:35.196 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7d858b6ea2875649,2d6542b6f051738b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=7d858b6ea2875649, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:06:35.196 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,7d858b6ea2875649,2d6542b6f051738b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=7d858b6ea2875649, duration=1 ms] 2024-10-16 14:06:35.275 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_thread_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703043730102882304 2024-10-16 14:06:35.275 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_memory_rss_bytes|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703043730098688000 2024-10-16 14:06:35.275 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_uptime_seconds|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703043730111270912 2024-10-16 14:06:35.275 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_exists|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703043730086105088 2024-10-16 14:06:35.275 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_fd_count|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703043730090299392 2024-10-16 14:06:35.275 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_count|app=HOST|env_type=host|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703043730069327872 2024-10-16 14:06:35.332 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdKeyService : Clean series info in cache, key=process_cpu_percent|app=HOST|name=obproxy|obproxy_cluster=xc_OB43|obproxy_cluster_id=1000002|obproxy_work_mode=configUrl|svr_ip=172.28.46.151, id=703043730077716480 2024-10-16 14:06:35.625 INFO 4486 --- [ocp-async-1,05d3ee803c5ed68c,831f5f302c355c14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:36.240 INFO 4486 --- [ocp-schedules-19,5ddb91722dbcfdd4,bbc66b4d90b8a8a6] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=39968 2024-10-16 14:06:37.063 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:06:37.064 INFO 4486 --- [ocp-schedules-20,3632b012ffcff04a,6f32c6f9b9117885] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:06:37.066 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:06:37.067 INFO 4486 --- [ocp-schedules-4,84bff4d51bf36cca,64afe01ec9416c56] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:06:37.068 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:06:37.069 INFO 4486 --- [ocp-schedules-4,84bff4d51bf36cca,64afe01ec9416c56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:37.070 INFO 4486 --- [ocp-schedules-20,3632b012ffcff04a,6f32c6f9b9117885] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:06:37.081 INFO 4486 --- [ocp-schedules-11,78ff33961fab3e25,4dc24178446087ca] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:06:37.083 INFO 4486 --- [ocp-schedules-15,215f846115aeb3a6,98c4ff881d9974a9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2024-10-16 14:06:37.083 INFO 4486 --- [ocp-schedules-11,78ff33961fab3e25,4dc24178446087ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:06:37.085 INFO 4486 --- [ocp-schedules-15,215f846115aeb3a6,98c4ff881d9974a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:37.090 INFO 4486 --- [ocp-schedules-30,d9d10975a4480ac9,fda02c2034e90fc7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:06:37.091 INFO 4486 --- [host-check-agent-status1,d9d10975a4480ac9,64265fef73afc3bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:06:37.091 INFO 4486 --- [host-check-agent-status2,d9d10975a4480ac9,2911d48bfa529416] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:06:37.092 INFO 4486 --- [host-check-agent-status3,d9d10975a4480ac9,ffd6d23aed6df4bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:06:37.092 INFO 4486 --- [ocp-schedules-18,fdcb7a17ca6c7efb,ab1224fc5b907441] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:06:37.092 INFO 4486 --- [host-check-agent-status4,d9d10975a4480ac9,6c1b27a8f057750e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:06:37.092 INFO 4486 --- [ocp-schedules-19,8b65f0b3f78b3c96,ee0b5d4779d6fd92] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:06:37.092 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:06:37.093 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:06:37.095 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:06:37.097 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:06:37.097 INFO 4486 --- [host-check-clockdiff3,8b65f0b3f78b3c96,bca92abd6fc1510d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:06:37.097 INFO 4486 --- [host-check-clockdiff2,8b65f0b3f78b3c96,1aea3d3ea407f6c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:06:37.098 INFO 4486 --- [host-check-clockdiff1,8b65f0b3f78b3c96,17cdb48363a5ea1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:06:37.098 INFO 4486 --- [host-check-clockdiff4,8b65f0b3f78b3c96,e44a888b69792b28] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:06:37.101 INFO 4486 --- [host-check-agent-status4,d9d10975a4480ac9,6c1b27a8f057750e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:06:37.102 INFO 4486 --- [host-check-agent-status1,d9d10975a4480ac9,64265fef73afc3bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:37.102 INFO 4486 --- [host-check-agent-status3,d9d10975a4480ac9,ffd6d23aed6df4bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:06:37.103 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:37.103 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:06:37.113 INFO 4486 --- [host-check-agent-status3,d9d10975a4480ac9,ffd6d23aed6df4bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:06:37.113 INFO 4486 --- [host-check-agent-status4,d9d10975a4480ac9,6c1b27a8f057750e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:06:37.113 INFO 4486 --- [host-check-agent-status2,d9d10975a4480ac9,2911d48bfa529416] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:06:37.113 INFO 4486 --- [host-check-agent-status1,d9d10975a4480ac9,64265fef73afc3bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:37.117 INFO 4486 --- [host-check-clockdiff1,8b65f0b3f78b3c96,17cdb48363a5ea1d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058797 0 0 , error= 2024-10-16 14:06:37.130 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:06:37.130 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:06:37.117 INFO 4486 --- [host-check-clockdiff2,8b65f0b3f78b3c96,1aea3d3ea407f6c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058797 4 4 , error= 2024-10-16 14:06:37.134 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:06:37.134 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:37.143 INFO 4486 --- [host-check-clockdiff4,8b65f0b3f78b3c96,e44a888b69792b28] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058797 4 4 , error= 2024-10-16 14:06:37.143 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:06:37.143 INFO 4486 --- [host-check-agent-status2,d9d10975a4480ac9,2911d48bfa529416] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:06:37.144 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:06:37.146 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.147 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:37.148 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:06:37.153 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.154 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 14:06:37.167 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,080df180db3d9140,adddf7293dac53e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&ObClusterId=1701313637&version=2, client=10.197.49.184, traceId=080df180db3d9140, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=OceanBase,1701313637,false,2,] 2024-10-16 14:06:37.168 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,080df180db3d9140,adddf7293dac53e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.184, traceId=080df180db3d9140, duration=2 ms] 2024-10-16 14:06:37.170 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:37.171 INFO 4486 --- [host-check-agent-status4,d9d10975a4480ac9,6c1b27a8f057750e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:06:37.171 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.172 INFO 4486 --- [host-check-agent-status2,d9d10975a4480ac9,2911d48bfa529416] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:06:37.172 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 14:06:37.177 INFO 4486 --- [host-check-agent-status3,d9d10975a4480ac9,ffd6d23aed6df4bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:06:37.178 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:06:37.178 INFO 4486 --- [host-check-agent-status1,d9d10975a4480ac9,64265fef73afc3bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:06:37.178 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.178 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:06:37.179 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:06:37.179 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.180 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.180 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.180 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:06:37.180 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2024-10-16 14:06:37.181 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.181 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:37.181 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:06:37.184331228+08:00 2024-10-16 14:06:37.181 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:06:37.184569914+08:00 2024-10-16 14:06:37.181 INFO 4486 --- [host-check-agent-status4,d9d10975a4480ac9,6c1b27a8f057750e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:06:37.181 INFO 4486 --- [host-check-agent-status4,d9d10975a4480ac9,6c1b27a8f057750e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:06:37.181 INFO 4486 --- [host-check-agent-status4,d9d10975a4480ac9,6c1b27a8f057750e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:06:37.181 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:06:37.181+08:00 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:06:37.181+08:00 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:06:37.186138355+08:00 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-agent-status3,d9d10975a4480ac9,ffd6d23aed6df4bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-agent-status3,d9d10975a4480ac9,ffd6d23aed6df4bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-agent-status3,d9d10975a4480ac9,ffd6d23aed6df4bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:06:37.182 INFO 4486 --- [host-check-agent-status2,d9d10975a4480ac9,2911d48bfa529416] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:06:37.183 INFO 4486 --- [host-check-agent-status2,d9d10975a4480ac9,2911d48bfa529416] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:06:37.183 INFO 4486 --- [host-check-agent-status2,d9d10975a4480ac9,2911d48bfa529416] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:06:37.183 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:06:37.183 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:06:37.182+08:00 2024-10-16 14:06:37.183 INFO 4486 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.197.49.184, 2882] 2024-10-16 14:06:37.184 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:06:37.184027069+08:00 2024-10-16 14:06:37.184 INFO 4486 --- [host-check-agent-status1,d9d10975a4480ac9,64265fef73afc3bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:06:37.184 INFO 4486 --- [host-check-agent-status1,d9d10975a4480ac9,64265fef73afc3bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:06:37.184 INFO 4486 --- [host-check-agent-status1,d9d10975a4480ac9,64265fef73afc3bb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:06:37.185 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:06:37.184+08:00 2024-10-16 14:06:37.185 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:06:37.181+08:00 and status AVAILABLE 2024-10-16 14:06:37.185 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:06:37.181+08:00 and status AVAILABLE 2024-10-16 14:06:37.185 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:06:37.185 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:06:37.182+08:00 and status AVAILABLE 2024-10-16 14:06:37.187 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:06:37.184+08:00 and status AVAILABLE 2024-10-16 14:06:37.187 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:06:37.187 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:06:37.188 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:06:37.189 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:06:37.191 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.193 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.193 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.193 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.193 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.193 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.194 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.195 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:06:37.196 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:06:37.196 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:06:37.196 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:06:37.196 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:37.197 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.197 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:06:37.197 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:06:37.198 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2024-10-16 14:06:37.198 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:06:37.198 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:37.203 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.204 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:37.209 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.213 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.152, 2882] 2024-10-16 14:06:37.213 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:06:37.216 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.216 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.217 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:37.220 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.151, 2882] 2024-10-16 14:06:37.221 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:37.221 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:37.221 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:37.221 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:06:37.223 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.226 INFO 4486 --- [Check root service 3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [172.28.46.153, 2882] 2024-10-16 14:06:37.230 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.230 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:06:37.234 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.235 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:06:37.244 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.245 INFO 4486 --- [ocp-schedules-21,8f09082b26dc74e9,7a053df555a68729] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:06:37.250 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:06:37.253 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:06:37.261 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:06:37.267 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:06:37.271 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:06:37.273 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:06:37.280 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:37.282 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:06:37.283 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:37.285 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:06:37.298 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:37.301 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:06:37.308 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:06:37.314 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:06:37.320 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:06:37.324 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:06:37.335 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:06:37.338 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:37.339 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:37.346 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:06:37.361 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:06:37.362 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:37.367 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:06:37.371 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:06:37.372 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:06:37.377 INFO 4486 --- [host-check-host-status3,fdcb7a17ca6c7efb,b6c08c7d2c264388] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:06:37.382 INFO 4486 --- [host-check-host-status4,fdcb7a17ca6c7efb,052e1801d18a13e9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:06:37.390 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:06:37.398 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:06:37.409 INFO 4486 --- [host-check-host-status2,fdcb7a17ca6c7efb,54aa44aa7f609b4a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:06:37.436 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:37.439 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.440 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:06:37.441 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.442 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:06:37.442 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.443 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:06:37.454 INFO 4486 --- [host-check-host-status1,fdcb7a17ca6c7efb,054ec19934c906e2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:06:37.455 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.455 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.455 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.455 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.455 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.455 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.483 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:06:37.520 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:06:37.555 INFO 4486 --- [ocp-schedules-2,3e6655e341c2838f,985e71f39f5de93b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:06:37.580 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:37.598 WARN 4486 --- [scanner-13332908-1,3e6655e341c2838f,c45bdf55c55eb173] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:06:37.610 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:37.614 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.615 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:06:37.618 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:06:37.619 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.620 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:06:37.621 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.622 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.622 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:37.623 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:06:37.626 INFO 4486 --- [ocp-async-10,755ed7bb9231707c,c64ac6706dea90be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:37.638 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.638 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.638 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.638 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.638 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.638 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.647 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:06:37.929 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:06:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:06:37.976 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:37.979 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.980 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:06:37.981 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.981 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.982 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:06:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:06:37.992 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.992 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.992 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.992 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.992 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:37.993 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:37.994 INFO 4486 --- [prometheus-minute-schedule-1,cf9ba42fd1f90506,ca35af2523975794] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:37.998 INFO 4486 --- [ocp-schedules-23,8e80f584c607943a,2895d45533c37bfb] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2024-10-16 14:06:37.998 INFO 4486 --- [ocp-schedules-23,8e80f584c607943a,2895d45533c37bfb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2024-10-16 14:06:38.000 INFO 4486 --- [prometheus-minute-schedule-1,cf9ba42fd1f90506,ca35af2523975794] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:38.012 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:06:38.031 INFO 4486 --- [pool-10-thread-6,f037455783c53da6,aefb86103ff75a84] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2024-10-16 14:06:38.035 INFO 4486 --- [pool-10-thread-5,f037455783c53da6,6fb9988f5f1c4b40] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2024-10-16 14:06:38.035 INFO 4486 --- [pool-10-thread-8,f037455783c53da6,31e94caec8ce72e0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2024-10-16 14:06:38.035 INFO 4486 --- [pool-10-thread-5,f037455783c53da6,6fb9988f5f1c4b40] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2024-10-16 14:06:38.036 INFO 4486 --- [pool-10-thread-7,f037455783c53da6,486b05a16ced3617] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2024-10-16 14:06:38.039 INFO 4486 --- [pool-10-thread-6,f037455783c53da6,aefb86103ff75a84] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2024-10-16 14:06:38.039 INFO 4486 --- [pool-10-thread-6,f037455783c53da6,aefb86103ff75a84] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2024-10-16 14:06:38.041 INFO 4486 --- [pool-10-thread-5,f037455783c53da6,6fb9988f5f1c4b40] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2024-10-16 14:06:38.042 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of obce_3zones 2024-10-16 14:06:38.042 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of obce_3zones 2024-10-16 14:06:38.042 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of glob_3zone 2024-10-16 14:06:38.042 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of glob_3zone 2024-10-16 14:06:38.048 INFO 4486 --- [ForkJoinPool-6-worker-7,f037455783c53da6,a42717d60dafcbeb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of OceanBase 2024-10-16 14:06:38.048 INFO 4486 --- [ForkJoinPool-6-worker-7,f037455783c53da6,a42717d60dafcbeb] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of OceanBase 2024-10-16 14:06:38.048 INFO 4486 --- [pool-10-thread-5,f037455783c53da6,6fb9988f5f1c4b40] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2024-10-16 14:06:38.057 INFO 4486 --- [host-check-clockdiff3,8b65f0b3f78b3c96,bca92abd6fc1510d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058798 4 4 , error= 2024-10-16 14:06:38.064 INFO 4486 --- [pool-overview-container-3,270b021c17065b3c,e5160d46cde292cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.066 INFO 4486 --- [pool-overview-container-3,270b021c17065b3c,e5160d46cde292cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 14:06:38.075 INFO 4486 --- [ForkJoinPool-6-worker-7,f037455783c53da6,a42717d60dafcbeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.076 INFO 4486 --- [ForkJoinPool-6-worker-7,f037455783c53da6,a42717d60dafcbeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:06:38.077 INFO 4486 --- [ForkJoinPool-6-worker-7,f037455783c53da6,a42717d60dafcbeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.078 INFO 4486 --- [ForkJoinPool-6-worker-7,f037455783c53da6,a42717d60dafcbeb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:06:38.081 INFO 4486 --- [ForkJoinPool-6-worker-5,f037455783c53da6,8507edd9bbd4f1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.083 INFO 4486 --- [ForkJoinPool-6-worker-5,f037455783c53da6,8507edd9bbd4f1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:06:38.085 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.086 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:06:38.088 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.091 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:06:38.093 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.093 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:06:38.094 WARN 4486 --- [ForkJoinPool-6-worker-7,f037455783c53da6,a42717d60dafcbeb] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :OceanBase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=OceanBase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:166) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:143) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$668a737d.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:242) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:214) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:67) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$9(OasStatServiceImpl.java:278) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163) 2024-10-16 14:06:38.095 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.095 INFO 4486 --- [ForkJoinPool-6-worker-5,f037455783c53da6,8507edd9bbd4f1dc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=ocp_meta, dbUser=root 2024-10-16 14:06:38.095 INFO 4486 --- [pool-overview-container-3,270b021c17065b3c,e5160d46cde292cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.095 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:06:38.096 INFO 4486 --- [ForkJoinPool-6-worker-5,f037455783c53da6,8507edd9bbd4f1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.098 INFO 4486 --- [ForkJoinPool-6-worker-5,f037455783c53da6,8507edd9bbd4f1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:06:38.098 INFO 4486 --- [pool-overview-container-3,270b021c17065b3c,e5160d46cde292cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2024-10-16 14:06:38.099 INFO 4486 --- [ForkJoinPool-6-worker-5,f037455783c53da6,8507edd9bbd4f1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.100 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.101 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:06:38.101 INFO 4486 --- [ForkJoinPool-6-worker-5,f037455783c53da6,8507edd9bbd4f1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:06:38.101 INFO 4486 --- [ForkJoinPool-6-worker-12,f037455783c53da6,4e126bd8c712558d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.103 INFO 4486 --- [ForkJoinPool-6-worker-12,f037455783c53da6,4e126bd8c712558d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:06:38.105 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.110 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 14:06:38.111 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 14:06:38.113 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.114 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:06:38.120 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.121 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:06:38.124 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.125 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:06:38.131 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.131 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:06:38.132 INFO 4486 --- [ForkJoinPool-6-worker-12,f037455783c53da6,4e126bd8c712558d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 14:06:38.136 INFO 4486 --- [ForkJoinPool-6-worker-12,f037455783c53da6,4e126bd8c712558d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.138 INFO 4486 --- [ForkJoinPool-6-worker-12,f037455783c53da6,4e126bd8c712558d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:06:38.143 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.143 INFO 4486 --- [ForkJoinPool-6-worker-12,f037455783c53da6,4e126bd8c712558d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.144 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:06:38.145 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:38.145 INFO 4486 --- [ForkJoinPool-6-worker-12,f037455783c53da6,4e126bd8c712558d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:06:38.149 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.154 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:06:38.154 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.157 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:38.158 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.160 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 14:06:38.160 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.163 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:06:38.166 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2024-10-16 14:06:38.169 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.170 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:06:38.171 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.176 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.178 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:06:38.178 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 14:06:38.179 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 14:06:38.180 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.182 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:06:38.183 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of obce_3zones 2024-10-16 14:06:38.186 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.188 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:06:38.201 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=root 2024-10-16 14:06:38.202 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.203 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2024-10-16 14:06:38.212 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.212 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.213 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2024-10-16 14:06:38.214 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:06:38, 2024-10-16 14:06:38] 2024-10-16 14:06:38.216 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.217 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of glob_3zone 2024-10-16 14:06:38.219 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.221 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1010] 2024-10-16 14:06:38.221 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.222 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2024-10-16 14:06:38.228 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1004] 2024-10-16 14:06:38.230 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:38.231 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.238 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu01, dbUser=root 2024-10-16 14:06:38.238 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 14:06:38.239 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.241 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 14:06:38.241 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:06:38, 2024-10-16 14:06:38] 2024-10-16 14:06:38.242 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.246 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu01, dbUser=root 2024-10-16 14:06:38.246 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2024-10-16 14:06:38.248 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.249 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.249 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:06:38, 2024-10-16 14:06:38] 2024-10-16 14:06:38.250 INFO 4486 --- [ForkJoinPool-6-worker-3,f037455783c53da6,e81e34fcd489fa37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2024-10-16 14:06:38.253 INFO 4486 --- [pool-10-thread-8,f037455783c53da6,31e94caec8ce72e0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2024-10-16 14:06:38.255 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:06:38.268 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=yewu02, dbUser=root 2024-10-16 14:06:38.270 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.271 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:38.272 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:06:38, 2024-10-16 14:06:38] 2024-10-16 14:06:38.273 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.279 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of obce_3zones 2024-10-16 14:06:38.279 INFO 4486 --- [ForkJoinPool-6-worker-14,f037455783c53da6,9070c8c43807b7e4] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of obce_3zones 2024-10-16 14:06:38.280 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1010] 2024-10-16 14:06:38.283 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.283 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=yewu02, dbUser=root 2024-10-16 14:06:38.285 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.286 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:06:38, 2024-10-16 14:06:38] 2024-10-16 14:06:38.288 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2024-10-16 14:06:38.307 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:38.309 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.310 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:06:38.311 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:38.312 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.313 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:06:38.320 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=root 2024-10-16 14:06:38.322 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:38.322 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:38.322 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:38.322 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:38.322 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:06:38.322 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.322 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.324 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2024-10-09 14:06:38, 2024-10-16 14:06:38] 2024-10-16 14:06:38.325 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:38.327 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:38.327 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of glob_3zone 2024-10-16 14:06:38.327 INFO 4486 --- [ForkJoinPool-6-worker-0,f037455783c53da6,e0b952f02a5eafee] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of glob_3zone 2024-10-16 14:06:38.332 INFO 4486 --- [pool-10-thread-7,f037455783c53da6,486b05a16ced3617] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2024-10-16 14:06:38.340 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1004] 2024-10-16 14:06:38.402 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:06:38.580 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,bbe35f25baf493fd,60c3b31327c53b2e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=bbe35f25baf493fd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:38.581 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,bbe35f25baf493fd,60c3b31327c53b2e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=bbe35f25baf493fd, duration=2 ms] 2024-10-16 14:06:38.607 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:06:38.406046, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:06:38.607 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:06:38.406046, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:06:38.645 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:06:38.406046} } 2024-10-16 14:06:38.699 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:06:38.706 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:06:38.710 INFO 4486 --- [ocp-schedules-22,8a808d5744c89d38,89ecf8f772b561c8] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:06:39.626 INFO 4486 --- [ocp-async-2,21bf7941da7d2532,d368f45e48319c6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:39.634 INFO 4486 --- [ocp-schedules-22,5ba6286d37efd900,b31a3aeef09e9f58] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:06:40.008 INFO 4486 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:06:40.008 INFO 4486 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:06:40.030 INFO 4486 --- [Check obproxy2,2cfca55783f9b8ae,5deda9def83016b7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:06:40.030 INFO 4486 --- [Check obproxy1,990a0854e9b13d67,80011bbe45cf4367] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:06:40.030 INFO 4486 --- [Check obproxy2,2cfca55783f9b8ae,5deda9def83016b7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:40.030 INFO 4486 --- [Check obproxy1,990a0854e9b13d67,80011bbe45cf4367] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:40.031 INFO 4486 --- [Collect obproxy1,20944b18f1571d5a,df5a6ea42efd819b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:06:40.031 INFO 4486 --- [Collect obproxy1,20944b18f1571d5a,df5a6ea42efd819b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:40.032 INFO 4486 --- [Collect obproxy2,0528942f38995745,c024420266ebf3ff] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:06:40.033 INFO 4486 --- [Collect obproxy2,0528942f38995745,c024420266ebf3ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:06:40.055 INFO 4486 --- [Check obproxy2,2cfca55783f9b8ae,5deda9def83016b7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:06:40.055 INFO 4486 --- [Check obproxy2,2cfca55783f9b8ae,5deda9def83016b7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:06:40.055 INFO 4486 --- [Collect obproxy2,0528942f38995745,c024420266ebf3ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:06:40.060 INFO 4486 --- [Collect obproxy2,0528942f38995745,c024420266ebf3ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:06:40.061 INFO 4486 --- [Collect obproxy1,20944b18f1571d5a,df5a6ea42efd819b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:06:40.065 INFO 4486 --- [Collect obproxy1,20944b18f1571d5a,df5a6ea42efd819b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:06:40.065 INFO 4486 --- [Collect obproxy2,0528942f38995745,c024420266ebf3ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:06:40.066 INFO 4486 --- [Check obproxy1,990a0854e9b13d67,80011bbe45cf4367] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:06:40.066 INFO 4486 --- [Check obproxy1,990a0854e9b13d67,80011bbe45cf4367] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:06:40.068 INFO 4486 --- [Collect obproxy1,20944b18f1571d5a,df5a6ea42efd819b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:06:40.599 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f5448265a08e3cc7,0984965fb54d9e97] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f5448265a08e3cc7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:40.599 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,f5448265a08e3cc7,0984965fb54d9e97] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f5448265a08e3cc7, duration=1 ms] 2024-10-16 14:06:40.623 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:40.994 INFO 4486 --- [prometheus-minute-schedule-1,6b7271d0fc2058d3,1a05ddc3d7b41941] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:40.995 INFO 4486 --- [prometheus-hour-schedule-1,7d1dcdb151470f2f,4a3d9bbb088ac4a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:06:41.001 INFO 4486 --- [prometheus-minute-schedule-1,6b7271d0fc2058d3,1a05ddc3d7b41941] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:41.476 INFO 4486 --- [ocp-schedules-17,c6753ae731968b4f,67a1abc63300e285] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:06:41.626 INFO 4486 --- [ocp-async-4,be79a7c8e21714f0,4a490f67a5fed08d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:42.161 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c2362906ecee8785,db1f63e04a0831db] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c2362906ecee8785, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:42.161 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,c2362906ecee8785,db1f63e04a0831db] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c2362906ecee8785, duration=1 ms] 2024-10-16 14:06:42.202 INFO 4486 --- [ocp-schedules-8,95bb18f31efb0206,42fc429e5664a4dd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:06:42.451 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4904e9dc5fbca782,3b2421bc849edfb1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4904e9dc5fbca782, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:42.452 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,4904e9dc5fbca782,3b2421bc849edfb1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4904e9dc5fbca782, duration=1 ms] 2024-10-16 14:06:43.035 INFO 4486 --- [ocp-async-9,5d9b67a366924e59,12e5b4fe4d367e3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:06:43.624 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:43.626 INFO 4486 --- [ocp-async-8,1429a365dcb4bd7e,f04770500f8156c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:43.994 INFO 4486 --- [prometheus-minute-schedule-1,eac3cf15983517e8,b61ab8c152cea8b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:44.000 INFO 4486 --- [prometheus-minute-schedule-1,eac3cf15983517e8,b61ab8c152cea8b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:45.625 INFO 4486 --- [ocp-schedules-13,875166b1430c3638,06df717618b0fafd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:06:45.626 INFO 4486 --- [ocp-async-9,86b10ae1a3c1ae92,47ae76efcdf0038a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:45.630 INFO 4486 --- [ocp-schedules-13,875166b1430c3638,06df717618b0fafd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:06:45.630 INFO 4486 --- [ocp-schedules-13,875166b1430c3638,06df717618b0fafd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:06:45.659 INFO 4486 --- [ocp-schedules-10,6ae08c47adcffec6,aac1f19ca1b168a4] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:06:45.659 INFO 4486 --- [ocp-schedules-10,6ae08c47adcffec6,aac1f19ca1b168a4] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:06:46.626 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:46.749 INFO 4486 --- [ocp-schedules-16,bc42ab9e7edc088f,0805a0222418c472] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:06:46.750 INFO 4486 --- [ocp-schedules-16,07e9bf65a67aeb9b,c6775be5268041d4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:06:46.751 INFO 4486 --- [ocp-schedules-16,07e9bf65a67aeb9b,c6775be5268041d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:06:46.777 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.778 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:46.788 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.789 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:46.816 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.816 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:46.817 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:46.817 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.817 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:06:46.849 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:46.850 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.850 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:46.852 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:06:46.852 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.855 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:06:46.857 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:46.868 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:46.868 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:46.879 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:46.889 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:46.890 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.891 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:46.891 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:06:46.896 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:06:46.901 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:46.903 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:46.915 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:46.917 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.918 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:06:46.926 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:06:46.956 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:06:46.956 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:06:46.966 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:46.968 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.968 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:06:46.972 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:46.975 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:06:46.978 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:46.994 INFO 4486 --- [prometheus-minute-schedule-1,cf7621185db0b255,4064f494e213e13a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:46.999 INFO 4486 --- [prometheus-minute-schedule-1,cf7621185db0b255,4064f494e213e13a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:47.003 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:06:47.004 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:47.004 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:47.015 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:47.016 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:47.017 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:06:47.027 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:06:47.027 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:06:47.036 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:47.036 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:47.060 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:47.060 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:47.071 INFO 4486 --- [ocp-schedules-4,84bff4d51bf36cca,64afe01ec9416c56] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:06:47.086 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:47.086 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:06:47.113 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:47.113 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:06:47.138 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:06:47.164 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:47.165 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:47.166 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:06:47.170 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:06:47.175 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:06:47.180 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:06:47.182 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:06:47.186 INFO 4486 --- [ocp-schedules-11,78ff33961fab3e25,4dc24178446087ca] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:06:47.205 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:06:47.229 INFO 4486 --- [ocp-schedules-15,215f846115aeb3a6,98c4ff881d9974a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=3, successCount=3 2024-10-16 14:06:47.316 INFO 4486 --- [ocp-schedules-7,6aba2e443262ce43,f93c3ad6bfe4abdb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:06:47.346 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:06:47.372 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:06:47.374 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:06:47.375 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:06:47.403 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:06:47.403 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:06:47.627 INFO 4486 --- [ocp-async-6,d2e31d58d84a55c2,6ec8f0481e51d1bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:47.732 INFO 4486 --- [ocp-schedules-2,085fe360b6f78a4c,90fcf047f53135bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:06:47.773 WARN 4486 --- [scanner-13333140-1,085fe360b6f78a4c,eec27820f32545fb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:06:47.870 INFO 4486 --- [alarm-detect-18,085fe360b6f78a4c,aef02ed260c88a24] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058775 ~ 1729058797 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058790, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058790, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058790=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:06:47.878 INFO 4486 --- [alarm-detect-18,085fe360b6f78a4c,aef02ed260c88a24] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 14:06:48.586 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,554db4e8650ff92c,d96723753c322775] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=554db4e8650ff92c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:48.586 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,554db4e8650ff92c,d96723753c322775] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=554db4e8650ff92c, duration=2 ms] 2024-10-16 14:06:49.627 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:49.627 INFO 4486 --- [ocp-async-5,9e1c3604acc2f594,3c98c2e640b26a23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:49.994 INFO 4486 --- [prometheus-minute-schedule-1,637ed61de17b2ac3,f09bc5b2c0026b89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:49.999 INFO 4486 --- [prometheus-minute-schedule-1,637ed61de17b2ac3,f09bc5b2c0026b89] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:50.603 INFO 4486 --- [ocp-schedules-25,4e38743b3aa1ddc1,60834e65a7903918] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:06:50.603 INFO 4486 --- [ocp-schedules-25,4e38743b3aa1ddc1,60834e65a7903918] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:06:50.604 INFO 4486 --- [ocp-schedules-25,4e38743b3aa1ddc1,60834e65a7903918] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:06:50.604 INFO 4486 --- [ocp-schedules-25,4e38743b3aa1ddc1,60834e65a7903918] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:06:50.605 INFO 4486 --- [ocp-schedules-25,4e38743b3aa1ddc1,60834e65a7903918] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:06:50.605 INFO 4486 --- [ocp-schedules-25,4e38743b3aa1ddc1,60834e65a7903918] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703043730111270912 2024-10-16 14:06:50.606 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,de7d026de15aa20d,4e5e62b3b072ffbb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=de7d026de15aa20d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:50.606 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,de7d026de15aa20d,4e5e62b3b072ffbb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=de7d026de15aa20d, duration=2 ms] 2024-10-16 14:06:50.607 INFO 4486 --- [ocp-schedules-25,4e38743b3aa1ddc1,60834e65a7903918] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=56, maxSyncedRecordId=703046246634950656 2024-10-16 14:06:50.927 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 14:06:50.927 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 14:06:50.940 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 14:06:50.940 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 14:06:50.940 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 14:06:50.951 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 14:06:50.951 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 14:06:50.951 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 14:06:50.957 INFO 4486 --- [ocp-schedules-5,4490379bbeac276f,84c7d428c72a54f3] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 14:06:51.482 INFO 4486 --- [ocp-schedules-14,e75264e1562669b8,ead9f61306a9f14a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:06:51.627 INFO 4486 --- [ocp-async-10,9d13fabb1076398d,856d0b2676161111] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:51.790 INFO 4486 --- [ocp-schedules-13,9c9d7cdc2b0a970f,4bd01975a865068b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:06:51.790 INFO 4486 --- [ocp-schedules-13,9c9d7cdc2b0a970f,4bd01975a865068b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:06:51.791 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:06:51.791 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:06:51.791 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:06:51.791 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:06:51.791 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:06:51.796 INFO 4486 --- [ocp-schedules-13,9c9d7cdc2b0a970f,4bd01975a865068b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:06:51.796 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:06:51.797 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:06:51.797 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:06:51.797 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:06:51.797 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:06:51.798 INFO 4486 --- [ocp-schedules-13,9c9d7cdc2b0a970f,4bd01975a865068b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:06:51.798 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:06:51.798 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:06:51.798 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:06:51.798 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:06:51.798 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:06:51.820 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:06:51.821 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:06:51.822 INFO 4486 --- [ocp-schedules-13,9c9d7cdc2b0a970f,4bd01975a865068b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:06:51.823 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:06:51.824 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:06:51.825 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:06:51.828 INFO 4486 --- [ocp-schedules-13,9c9d7cdc2b0a970f,4bd01975a865068b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:06:52.167 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,3fb2e23bf44283c4,e2c9982f982405bc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3fb2e23bf44283c4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:52.168 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,3fb2e23bf44283c4,e2c9982f982405bc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3fb2e23bf44283c4, duration=2 ms] 2024-10-16 14:06:52.456 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,7c98a6fe290c80a1,1e4318b8dc8b0cb3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7c98a6fe290c80a1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:52.456 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,7c98a6fe290c80a1,1e4318b8dc8b0cb3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7c98a6fe290c80a1, duration=1 ms] 2024-10-16 14:06:52.628 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:52.994 INFO 4486 --- [prometheus-minute-schedule-1,9d0cf66413f1ee8d,b8a3d58639862f5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:53.001 INFO 4486 --- [prometheus-minute-schedule-1,9d0cf66413f1ee8d,b8a3d58639862f5f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:53.035 INFO 4486 --- [ocp-async-8,c3f6cf61dc88235f,4472b3e8c2f45e0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:06:53.628 INFO 4486 --- [ocp-async-6,a86f13fd4ffd2b30,d02cdff79fee8296] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:55.628 INFO 4486 --- [ocp-async-7,6754a94316bf14e0,ecc6169aae0638de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:55.629 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:55.994 INFO 4486 --- [prometheus-minute-schedule-1,c6f3d0465b332058,b6d05d5d75991255] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:56.001 INFO 4486 --- [prometheus-minute-schedule-1,c6f3d0465b332058,b6d05d5d75991255] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:57.628 INFO 4486 --- [ocp-async-9,45e7f5b0be8db1a5,baf0fdb0650c918f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:57.955 INFO 4486 --- [ocp-schedules-22,88ddd0f49bf3d303,aff30aa557b9749e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:06:58.000 WARN 4486 --- [scanner-13333459-1,88ddd0f49bf3d303,8d8c5b90f6673a6a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:06:58.056 INFO 4486 --- [alarm-detect-10,88ddd0f49bf3d303,93864f7fd1d53eca] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058790 ~ 1729058808 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058805, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058805, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058805=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:06:58.591 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,60e9bbdf2b89910e,14473ab3bf75b596] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=60e9bbdf2b89910e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:06:58.591 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,60e9bbdf2b89910e,14473ab3bf75b596] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=60e9bbdf2b89910e, duration=2 ms] 2024-10-16 14:06:58.630 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:06:58.994 INFO 4486 --- [prometheus-minute-schedule-1,c5409090c096e80e,060a5f6e128f2294] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058820, interval=60 2024-10-16 14:06:59.001 INFO 4486 --- [prometheus-minute-schedule-1,c5409090c096e80e,060a5f6e128f2294] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:06:59.628 INFO 4486 --- [ocp-async-3,43b792394e604134,93d67f8f740ac186] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:06:59.638 INFO 4486 --- [ocp-schedules-9,6b58387d16891026,e865dcd308e6b42e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:07:00.611 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e18f6279ece92fac,682dfe8b75a1b064] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e18f6279ece92fac, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:00.611 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e18f6279ece92fac,682dfe8b75a1b064] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e18f6279ece92fac, duration=1 ms] 2024-10-16 14:07:01.488 INFO 4486 --- [ocp-schedules-13,b93b98ebdcf0ec28,9296fd0c05558d14] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:07:01.629 INFO 4486 --- [ocp-async-6,b2c582540c1d205f,e1b68903c2aa43b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:01.632 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:01.994 INFO 4486 --- [prometheus-minute-schedule-1,e1474212c08c5514,43548dbe5c66bea0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:01.999 INFO 4486 --- [prometheus-minute-schedule-1,e1474212c08c5514,43548dbe5c66bea0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:02.172 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,343ba45852c58bdd,af13dbf22849f7b7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=343ba45852c58bdd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:02.172 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,343ba45852c58bdd,af13dbf22849f7b7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=343ba45852c58bdd, duration=1 ms] 2024-10-16 14:07:02.461 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6b2be027bf37caf8,b150e0674f635208] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6b2be027bf37caf8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:02.461 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,6b2be027bf37caf8,b150e0674f635208] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6b2be027bf37caf8, duration=1 ms] 2024-10-16 14:07:03.011 INFO 4486 --- [ocp-schedules-6,77c6a3305f1eadce,e17190260af07a83] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:07:03.011+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:07:03.014 INFO 4486 --- [ocp-schedules-6,77c6a3305f1eadce,e17190260af07a83] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:07:03.024 INFO 4486 --- [ocp-schedules-6,77c6a3305f1eadce,e17190260af07a83] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:07:03.035 INFO 4486 --- [ocp-schedules-6,77c6a3305f1eadce,e17190260af07a83] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:07:03.035 INFO 4486 --- [ocp-async-1,c27db03d99995818,0dde52e14ddb6d10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:07:03.595 INFO 4486 --- [ocp-schedules-13,64522eb6d34f4bbc,4aacc17dc071365a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:07:03.596 INFO 4486 --- [ocp-schedules-13,64522eb6d34f4bbc,4aacc17dc071365a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:03.597 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:07:03.597 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:07:03.598 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:07:03.629 INFO 4486 --- [ocp-async-2,de122460627c295e,d6fc9db5f830423a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:04.633 INFO 4486 --- [pool-12-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:04.994 INFO 4486 --- [prometheus-minute-schedule-1,c4bfacd2db4b45c1,41a82c957b9e9432] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:05.001 INFO 4486 --- [prometheus-minute-schedule-1,c4bfacd2db4b45c1,41a82c957b9e9432] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:05.629 INFO 4486 --- [ocp-async-1,253c658a9c733264,504aeebf31b91e48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:07.066 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:07:07.067 INFO 4486 --- [ocp-schedules-24,aa2f23fe6a98bf80,fb30f154d341c576] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:07:07.069 INFO 4486 --- [ocp-schedules-24,aa2f23fe6a98bf80,fb30f154d341c576] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:07.081 INFO 4486 --- [ocp-schedules-17,158a4a92dbcb462e,30ca40690e18a2f7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:07:07.083 INFO 4486 --- [ocp-schedules-17,158a4a92dbcb462e,30ca40690e18a2f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:07.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:07:07.091 INFO 4486 --- [ocp-schedules-1,e373cc739a7bcd98,afbd4064e4d8e6a2] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:07:07.092 INFO 4486 --- [ocp-schedules-4,3758480fe163eeaf,8ec21fd351ef359d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:07:07.092 INFO 4486 --- [host-check-agent-status1,e373cc739a7bcd98,7a225def0a2c95c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:07:07.093 INFO 4486 --- [host-check-agent-status2,e373cc739a7bcd98,6d56797c236bc3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:07:07.093 INFO 4486 --- [host-check-agent-status3,e373cc739a7bcd98,b33aba74eb1ecc62] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:07:07.094 INFO 4486 --- [host-check-agent-status4,e373cc739a7bcd98,c5fdc042d8a42ca3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:07:07.094 INFO 4486 --- [ocp-schedules-27,ef5431f35703a702,f1ec4fa5e5b7dd8b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:07:07.094 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:07:07.093 INFO 4486 --- [host-check-clockdiff1,3758480fe163eeaf,460c1efb4c9e210f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:07:07.095 INFO 4486 --- [host-check-clockdiff2,3758480fe163eeaf,bce7b1df62c8cdc1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:07:07.096 INFO 4486 --- [host-check-clockdiff3,3758480fe163eeaf,048ff3ce929df44a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:07:07.096 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:07:07.096 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:07:07.101 INFO 4486 --- [host-check-clockdiff4,3758480fe163eeaf,ceffdc783afc1598] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:07:07.102 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:07:07.103 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:07.106 INFO 4486 --- [host-check-agent-status2,e373cc739a7bcd98,6d56797c236bc3d4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:07:07.107 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:07.107 INFO 4486 --- [host-check-agent-status1,e373cc739a7bcd98,7a225def0a2c95c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:07.107 INFO 4486 --- [host-check-agent-status4,e373cc739a7bcd98,c5fdc042d8a42ca3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:07:07.107 INFO 4486 --- [host-check-agent-status3,e373cc739a7bcd98,b33aba74eb1ecc62] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:07:07.108 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:07:07.108 INFO 4486 --- [host-check-agent-status2,e373cc739a7bcd98,6d56797c236bc3d4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:07:07.109 INFO 4486 --- [host-check-agent-status3,e373cc739a7bcd98,b33aba74eb1ecc62] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:07:07.110 INFO 4486 --- [host-check-agent-status1,e373cc739a7bcd98,7a225def0a2c95c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:07.110 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:07:07.118 INFO 4486 --- [host-check-agent-status4,e373cc739a7bcd98,c5fdc042d8a42ca3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:07:07.118 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:07:07.129 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:07:07.129 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:07:07.130 INFO 4486 --- [host-check-clockdiff3,3758480fe163eeaf,048ff3ce929df44a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058827 4 4 , error= 2024-10-16 14:07:07.133 INFO 4486 --- [host-check-clockdiff2,3758480fe163eeaf,bce7b1df62c8cdc1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058827 4 4 , error= 2024-10-16 14:07:07.134 INFO 4486 --- [host-check-clockdiff1,3758480fe163eeaf,460c1efb4c9e210f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058827 0 0 , error= 2024-10-16 14:07:07.137 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:07:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.142 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:07.143 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:07:07.153 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.154 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:07.158 INFO 4486 --- [host-check-agent-status1,e373cc739a7bcd98,7a225def0a2c95c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:07:07.159 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:07:07.160 INFO 4486 --- [host-check-agent-status4,e373cc739a7bcd98,c5fdc042d8a42ca3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:07:07.160 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:07:07.160 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:07:07.160 INFO 4486 --- [host-check-agent-status2,e373cc739a7bcd98,6d56797c236bc3d4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:07:07.161 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:07:07.162 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:07:07.161863620+08:00 2024-10-16 14:07:07.163 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:07:07.166844129+08:00 2024-10-16 14:07:07.163 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:07:07.162+08:00 2024-10-16 14:07:07.163 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:07:07.163+08:00 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:07:07.167974512+08:00 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status4,e373cc739a7bcd98,c5fdc042d8a42ca3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status1,e373cc739a7bcd98,7a225def0a2c95c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status4,e373cc739a7bcd98,c5fdc042d8a42ca3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status1,e373cc739a7bcd98,7a225def0a2c95c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status4,e373cc739a7bcd98,c5fdc042d8a42ca3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status1,e373cc739a7bcd98,7a225def0a2c95c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:07:07.166+08:00 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status3,e373cc739a7bcd98,b33aba74eb1ecc62] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:07:07.162+08:00 and status AVAILABLE 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:07:07.163+08:00 and status AVAILABLE 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status2,e373cc739a7bcd98,6d56797c236bc3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:07:07.166 INFO 4486 --- [host-check-agent-status2,e373cc739a7bcd98,6d56797c236bc3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:07:07.167 INFO 4486 --- [host-check-agent-status2,e373cc739a7bcd98,6d56797c236bc3d4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:07:07.168 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:07:07.169 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:07:07.169 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:07:07.166+08:00 and status AVAILABLE 2024-10-16 14:07:07.169 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:07:07.170 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:07:07.173981689+08:00 2024-10-16 14:07:07.170 INFO 4486 --- [host-check-agent-status3,e373cc739a7bcd98,b33aba74eb1ecc62] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:07:07.170 INFO 4486 --- [host-check-agent-status3,e373cc739a7bcd98,b33aba74eb1ecc62] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:07:07.170 INFO 4486 --- [host-check-agent-status3,e373cc739a7bcd98,b33aba74eb1ecc62] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:07:07.171 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:07:07.170+08:00 2024-10-16 14:07:07.171 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:07:07.174 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.174 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:07:07.170+08:00 and status AVAILABLE 2024-10-16 14:07:07.176 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:07:07.177 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:07.178 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:07:07.178 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:07.178 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:07:07.179 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:07:07.179 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:07:07.181 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:07:07.184 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:07:07.185 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:07:07.186 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.187 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:07:07.200 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.201 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:07:07.201 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:07.203 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:07.204 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:07.212 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.212 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:07.213 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:07:07.215 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.216 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:07:07.226 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:07.227 INFO 4486 --- [ocp-schedules-10,e39c280e703757ab,e2f0bd2e4f0f3797] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:07:07.232 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:07:07.242 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:07:07.243 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:07:07.246 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:07:07.253 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:07:07.258 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:07:07.260 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:07.261 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:07:07.269 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:07:07.270 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:07.280 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:07.285 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:07.290 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:07:07.299 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:07:07.302 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:07:07.313 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:07:07.315 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:07:07.317 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:07.328 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:07:07.333 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:07.340 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:07:07.343 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:07.346 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:07:07.354 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:07:07.358 INFO 4486 --- [host-check-host-status3,ef5431f35703a702,d32c7f7c0e31fc01] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:07:07.370 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:07:07.372 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:07.381 INFO 4486 --- [host-check-host-status4,ef5431f35703a702,940f19b3aa083601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:07:07.386 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:07:07.396 INFO 4486 --- [host-check-host-status2,ef5431f35703a702,40b79e6a56f061af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:07:07.425 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:07:07.436 INFO 4486 --- [host-check-host-status1,ef5431f35703a702,11bf2477fd5a36f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:07:07.629 INFO 4486 --- [ocp-async-8,2bdc2a7fc206593d,93b0b4de2b8f68d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:07.634 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:07.994 INFO 4486 --- [prometheus-minute-schedule-1,fc978a100802cbbc,f200ee4e3625d711] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:08.002 INFO 4486 --- [prometheus-minute-schedule-1,fc978a100802cbbc,f200ee4e3625d711] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:08.083 INFO 4486 --- [host-check-clockdiff4,3758480fe163eeaf,ceffdc783afc1598] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058828 4 4 , error= 2024-10-16 14:07:08.124 INFO 4486 --- [ocp-schedules-2,90aec1265368cfc9,fc0817844de19e2e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:07:08.201 WARN 4486 --- [scanner-13333712-1,90aec1265368cfc9,971b60b986609d6b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:07:08.597 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,19157b60f27823a7,9085ae0dd9d66912] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=19157b60f27823a7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:08.598 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,19157b60f27823a7,9085ae0dd9d66912] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=19157b60f27823a7, duration=3 ms] 2024-10-16 14:07:09.629 INFO 4486 --- [ocp-async-7,7279ae25868ec3bf,2427dc7ff925d963] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:10.006 INFO 4486 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:07:10.007 INFO 4486 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:07:10.028 INFO 4486 --- [Check obproxy1,b85149ba97acd804,47845b5ca493b483] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:07:10.028 INFO 4486 --- [Check obproxy2,55803ed7c090352f,56676b03a727774a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:07:10.028 INFO 4486 --- [Check obproxy2,55803ed7c090352f,56676b03a727774a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:10.028 INFO 4486 --- [Check obproxy1,b85149ba97acd804,47845b5ca493b483] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:10.030 INFO 4486 --- [Collect obproxy1,02600e6578856457,89085243c8cf6362] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:07:10.030 INFO 4486 --- [Collect obproxy1,02600e6578856457,89085243c8cf6362] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:10.032 INFO 4486 --- [Collect obproxy2,02fad21b62bb756d,9987522b6f83efce] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:07:10.032 INFO 4486 --- [Collect obproxy2,02fad21b62bb756d,9987522b6f83efce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:10.054 INFO 4486 --- [Check obproxy2,55803ed7c090352f,56676b03a727774a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:07:10.054 INFO 4486 --- [Check obproxy2,55803ed7c090352f,56676b03a727774a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:07:10.058 INFO 4486 --- [Collect obproxy2,02fad21b62bb756d,9987522b6f83efce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:07:10.058 INFO 4486 --- [Check obproxy1,b85149ba97acd804,47845b5ca493b483] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:07:10.058 INFO 4486 --- [Check obproxy1,b85149ba97acd804,47845b5ca493b483] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:07:10.060 INFO 4486 --- [Collect obproxy1,02600e6578856457,89085243c8cf6362] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:07:10.063 INFO 4486 --- [Collect obproxy2,02fad21b62bb756d,9987522b6f83efce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:07:10.065 INFO 4486 --- [Collect obproxy1,02600e6578856457,89085243c8cf6362] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:07:10.069 INFO 4486 --- [Collect obproxy2,02fad21b62bb756d,9987522b6f83efce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:07:10.070 INFO 4486 --- [Collect obproxy1,02600e6578856457,89085243c8cf6362] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:07:10.615 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f12a9792fbd6364c,8749d5812f91bfec] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f12a9792fbd6364c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:10.615 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f12a9792fbd6364c,8749d5812f91bfec] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f12a9792fbd6364c, duration=1 ms] 2024-10-16 14:07:10.635 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:10.978 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:07:10.994 INFO 4486 --- [prometheus-minute-schedule-1,b20ead6beb8f837c,0b5f380c3d601ced] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:11.000 INFO 4486 --- [prometheus-minute-schedule-1,b20ead6beb8f837c,0b5f380c3d601ced] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:07:11.494 INFO 4486 --- [ocp-schedules-19,a5388f382199e746,66dd4bee302a705e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:07:11.630 INFO 4486 --- [ocp-async-2,f56076927bc0b585,90656c93ce73ad7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:11.796 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058760, nodeCount=14, affectRows=14 2024-10-16 14:07:11.796 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058760, nodeCount=14, validCount=14 2024-10-16 14:07:11.825 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058760, nodeCount=491, affectRows=491 2024-10-16 14:07:11.828 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058760, nodeCount=489, validCount=489 2024-10-16 14:07:11.847 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058760, nodeCount=1000, affectRows=1000 2024-10-16 14:07:11.848 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058760, nodeCount=1000, affectRows=1000 2024-10-16 14:07:11.851 INFO 4486 --- [metric-archive-second-16,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058760, nodeCount=1000, validCount=1000 2024-10-16 14:07:11.852 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058760, nodeCount=999, validCount=999 2024-10-16 14:07:11.883 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058760, nodeCount=1000, affectRows=1000 2024-10-16 14:07:11.886 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058760, nodeCount=1000, validCount=1000 2024-10-16 14:07:12.176 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,3bf6c22335fd7cc7,586b1d031128254b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3bf6c22335fd7cc7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:12.176 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,3bf6c22335fd7cc7,586b1d031128254b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3bf6c22335fd7cc7, duration=1 ms] 2024-10-16 14:07:12.465 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,00e96283d5f7871e,d1d85474ecfce1bd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=00e96283d5f7871e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:12.465 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,00e96283d5f7871e,d1d85474ecfce1bd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=00e96283d5f7871e, duration=1 ms] 2024-10-16 14:07:13.036 INFO 4486 --- [ocp-async-9,d0581efb5e06ad4b,7f89d4dfe7423df0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:07:13.598 INFO 4486 --- [ocp-schedules-13,64522eb6d34f4bbc,4aacc17dc071365a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:07:13.630 INFO 4486 --- [ocp-async-6,a4c1c294c3b2a712,4f81e627c7b0ef55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:13.636 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:13.994 INFO 4486 --- [prometheus-minute-schedule-1,dd03c442cf6fea90,c14a2a669caba740] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:14.000 INFO 4486 --- [prometheus-minute-schedule-1,dd03c442cf6fea90,c14a2a669caba740] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:15.378 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058760, nodeCount=101, affectRows=101 2024-10-16 14:07:15.382 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058760, nodeCount=124, affectRows=124 2024-10-16 14:07:15.382 INFO 4486 --- [metric-archive-async-11,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058760, nodeCount=144, affectRows=144 2024-10-16 14:07:15.386 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058760, nodeCount=111, affectRows=111 2024-10-16 14:07:15.550 INFO 4486 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058760, nodeCount=111, validCount=111 2024-10-16 14:07:15.553 INFO 4486 --- [metric-archive-async-14,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058760, nodeCount=101, validCount=101 2024-10-16 14:07:15.580 INFO 4486 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058760, nodeCount=124, validCount=124 2024-10-16 14:07:15.605 INFO 4486 --- [metric-archive-async-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058760, nodeCount=144, validCount=144 2024-10-16 14:07:15.630 INFO 4486 --- [ocp-async-8,c7900c6dde49c586,6f04558571fc6760] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:15.909 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c2a3614cb9262161,3693dc8cb613e90f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=c2a3614cb9262161, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:07:15.910 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,c2a3614cb9262161,3693dc8cb613e90f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=c2a3614cb9262161, duration=2 ms] 2024-10-16 14:07:16.637 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:16.994 INFO 4486 --- [prometheus-minute-schedule-1,b9b3c34b16aac80e,a750e3ed356c82be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:17.000 INFO 4486 --- [prometheus-minute-schedule-1,b9b3c34b16aac80e,a750e3ed356c82be] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:17.071 INFO 4486 --- [ocp-schedules-24,aa2f23fe6a98bf80,fb30f154d341c576] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:07:17.159 INFO 4486 --- [ocp-schedules-17,158a4a92dbcb462e,30ca40690e18a2f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:07:17.318 INFO 4486 --- [ocp-schedules-26,3d407c39604973ab,f1379b84bc416019] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:07:17.404 INFO 4486 --- [ocp-schedules-16,07e9bf65a67aeb9b,c6775be5268041d4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:07:17.405 INFO 4486 --- [ocp-schedules-16,25e061e7672c1a7c,86646b85fecc2704] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:07:17.406 INFO 4486 --- [ocp-schedules-16,25e061e7672c1a7c,86646b85fecc2704] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:17.434 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.435 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:17.448 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.449 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:17.473 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:17.474 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.474 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:07:17.475 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.476 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:17.500 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:17.501 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.501 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:17.502 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:07:17.503 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.505 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:07:17.507 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:17.566 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.567 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:17.578 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.589 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:17.590 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.590 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:17.591 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:07:17.596 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:07:17.600 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:17.603 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:17.628 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:07:17.630 INFO 4486 --- [ocp-async-7,57f7e7cd79855233,c27d92f070b6516e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:17.634 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:17.636 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.637 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:07:17.680 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:07:17.681 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:07:17.685 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:17.687 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.687 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:17.691 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.694 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:07:17.698 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:17.702 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:07:17.712 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:17.713 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.714 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:07:17.723 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:07:17.723 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:07:17.723 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.723 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:17.747 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.747 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:17.770 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.770 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:17.801 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.801 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:17.827 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.827 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:17.851 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:17.875 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:17.877 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:17.878 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:17.882 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:07:17.887 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:07:17.891 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:07:17.893 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:07:17.914 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:07:18.069 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:07:18.095 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:18.096 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:18.099 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:07:18.126 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:07:18.126 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:07:18.354 INFO 4486 --- [ocp-schedules-13,a4600aec6d8e7755,0ee676322147e91b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:07:18.383 WARN 4486 --- [scanner-13334870-1,a4600aec6d8e7755,20efe50d778cff8b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:07:18.423 INFO 4486 --- [alarm-detect-13,a4600aec6d8e7755,aee5d6cc0635d3f6] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058805 ~ 1729058828 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058820, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058820, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058820=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:07:18.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,68f06181b9726de7,a0f1d2323233d62d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=68f06181b9726de7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:18.603 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,68f06181b9726de7,a0f1d2323233d62d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=68f06181b9726de7, duration=1 ms] 2024-10-16 14:07:18.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2ba29cb96709c4f1,987afdb1777ea6b6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=2ba29cb96709c4f1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:07:18.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,2ba29cb96709c4f1,987afdb1777ea6b6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=2ba29cb96709c4f1, duration=1 ms] 2024-10-16 14:07:19.630 INFO 4486 --- [ocp-async-1,59eeb66456a33512,31d826c3f507034d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:19.638 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:19.643 INFO 4486 --- [ocp-schedules-4,d89265ccf4d1353e,5ef227f965cd3a07] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:07:19.994 INFO 4486 --- [prometheus-minute-schedule-1,f336dbe4c5ca8307,092d84bc1ef780a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:20.001 INFO 4486 --- [prometheus-minute-schedule-1,f336dbe4c5ca8307,092d84bc1ef780a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:20.621 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,61a5dc4c3e12cc53,269ba0a358a87490] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=61a5dc4c3e12cc53, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:20.621 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,61a5dc4c3e12cc53,269ba0a358a87490] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=61a5dc4c3e12cc53, duration=2 ms] 2024-10-16 14:07:20.870 INFO 4486 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_PERF_DIAGNOSER, skipped 2024-10-16 14:07:20.870 INFO 4486 --- [pool-15-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:SQL_INDEX_DIAGNOSER, skipped 2024-10-16 14:07:20.996 INFO 4486 --- [prometheus-cache-monitor1,2abaa73ad6e7cf46,7921900d9c65356b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8187} 2024-10-16 14:07:20.996 INFO 4486 --- [prometheus-cache-monitor1,2abaa73ad6e7cf46,7921900d9c65356b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:07:21.499 INFO 4486 --- [ocp-schedules-18,6069b4da860a5a83,dd6369dc1838ee6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:07:21.631 INFO 4486 --- [ocp-async-6,0b40532f77a11af6,29260dbe2c4e5241] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:22.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,7b9fae7ff75a3759,9b5109aaf4332a7d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7b9fae7ff75a3759, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:22.182 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,7b9fae7ff75a3759,9b5109aaf4332a7d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7b9fae7ff75a3759, duration=1 ms] 2024-10-16 14:07:22.469 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,de519f99d23e5e45,8631898f9f57a5d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=de519f99d23e5e45, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:22.470 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,de519f99d23e5e45,8631898f9f57a5d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=de519f99d23e5e45, duration=2 ms] 2024-10-16 14:07:22.639 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:22.994 INFO 4486 --- [prometheus-minute-schedule-1,ef72eff9f552cbf5,b655e118235543da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:23.003 INFO 4486 --- [prometheus-minute-schedule-1,ef72eff9f552cbf5,b655e118235543da] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:23.036 INFO 4486 --- [ocp-async-5,042bfdbeea71cc62,e7414e07e5d6de75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:07:23.631 INFO 4486 --- [ocp-async-4,1cda7334aa402bb8,84a533f7cf8c8da1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:24.195 INFO 4486 --- [ocp-schedules-15,7c65fd0110ff7d60,bf6b10dbc6071593] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:07:25.414 INFO 4486 --- [ocp-schedules-7,92ec5ed94151c098,58bc7ecb1d47994d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:07:25.631 INFO 4486 --- [ocp-async-5,925ff35eb50f4294,87954661b1bb7e78] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:25.640 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:25.997 INFO 4486 --- [prometheus-minute-schedule-1,63ce88c7eb67e490,7591fd9ad978204d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:26.004 INFO 4486 --- [prometheus-minute-schedule-1,63ce88c7eb67e490,7591fd9ad978204d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:27.211 INFO 4486 --- [ocp-schedules-17,69d53676b41233e6,1cc015cadbfcac88] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:07:27.211 INFO 4486 --- [ocp-schedules-17,69d53676b41233e6,1cc015cadbfcac88] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:07:27.245 INFO 4486 --- [Collect all tenants major compactions1,69d53676b41233e6,bd51dff6fa4ceb8f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:07:27.246 INFO 4486 --- [Collect all tenants major compactions2,69d53676b41233e6,703414095af1d6dd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:07:27.246 INFO 4486 --- [Collect all tenants major compactions4,69d53676b41233e6,f5e376420fa62171] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:07:27.246 INFO 4486 --- [Collect all tenants major compactions5,69d53676b41233e6,86a80af77d7f61e8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:07:27.247 INFO 4486 --- [Collect all tenants major compactions3,69d53676b41233e6,8d5aafd28f64bc08] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:07:27.258 INFO 4486 --- [Collect all tenants major compactions1,69d53676b41233e6,bd51dff6fa4ceb8f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:27.258 INFO 4486 --- [Collect all tenants major compactions2,69d53676b41233e6,703414095af1d6dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:27.259 INFO 4486 --- [Collect all tenants major compactions1,69d53676b41233e6,bd51dff6fa4ceb8f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:07:27.259 INFO 4486 --- [Collect all tenants major compactions2,69d53676b41233e6,703414095af1d6dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:07:27.263 INFO 4486 --- [Collect all tenants major compactions2,69d53676b41233e6,703414095af1d6dd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:07:27.263 INFO 4486 --- [Collect all tenants major compactions1,69d53676b41233e6,bd51dff6fa4ceb8f] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:07:27.263 INFO 4486 --- [Collect all tenants major compactions2,69d53676b41233e6,703414095af1d6dd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:07:27.263 INFO 4486 --- [Collect all tenants major compactions1,69d53676b41233e6,bd51dff6fa4ceb8f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:07:27.264 INFO 4486 --- [Collect all tenants major compactions3,69d53676b41233e6,8d5aafd28f64bc08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:27.266 INFO 4486 --- [Collect all tenants major compactions3,69d53676b41233e6,8d5aafd28f64bc08] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:07:27.270 INFO 4486 --- [Collect all tenants major compactions3,69d53676b41233e6,8d5aafd28f64bc08] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:07:27.270 INFO 4486 --- [Collect all tenants major compactions3,69d53676b41233e6,8d5aafd28f64bc08] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:07:27.276 INFO 4486 --- [Collect all tenants major compactions5,69d53676b41233e6,86a80af77d7f61e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:27.276 INFO 4486 --- [Collect all tenants major compactions4,69d53676b41233e6,f5e376420fa62171] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:27.277 INFO 4486 --- [Collect all tenants major compactions5,69d53676b41233e6,86a80af77d7f61e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:07:27.278 INFO 4486 --- [Collect all tenants major compactions4,69d53676b41233e6,f5e376420fa62171] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:07:27.281 INFO 4486 --- [Collect all tenants major compactions4,69d53676b41233e6,f5e376420fa62171] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:07:27.281 INFO 4486 --- [Collect all tenants major compactions4,69d53676b41233e6,f5e376420fa62171] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:07:27.282 INFO 4486 --- [Collect all tenants major compactions5,69d53676b41233e6,86a80af77d7f61e8] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:07:27.282 INFO 4486 --- [Collect all tenants major compactions5,69d53676b41233e6,86a80af77d7f61e8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:07:27.284 INFO 4486 --- [ocp-schedules-17,69d53676b41233e6,1cc015cadbfcac88] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:07:27.631 INFO 4486 --- [ocp-async-9,de1e072a52b1c592,42d48a87fdd7487d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:28.076 INFO 4486 --- [ocp-schedules-20,e43ccc4e9b8c2cd8,4b92e7528a4b64ff] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:07:28.076+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:07:28.077 INFO 4486 --- [ocp-schedules-20,e43ccc4e9b8c2cd8,4b92e7528a4b64ff] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:07:28.078 INFO 4486 --- [ocp-schedules-20,e43ccc4e9b8c2cd8,4b92e7528a4b64ff] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:07:28.079 INFO 4486 --- [ocp-schedules-20,e43ccc4e9b8c2cd8,4b92e7528a4b64ff] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:07:28.487 INFO 4486 --- [ocp-schedules-14,3aa8834899456792,0e4ef0db5afec7a7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:07:28.515 WARN 4486 --- [scanner-13335089-1,3aa8834899456792,79f8f8fcc6503441] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:07:28.587 INFO 4486 --- [alarm-detect-17,3aa8834899456792,4aa6f28006519686] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058820 ~ 1729058838 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058835, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058835, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058835=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:07:28.611 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,88f3ad098d9718f9,04bae4cde9bf1e33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=88f3ad098d9718f9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:28.612 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,88f3ad098d9718f9,04bae4cde9bf1e33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=88f3ad098d9718f9, duration=5 ms] 2024-10-16 14:07:28.647 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:28.994 INFO 4486 --- [prometheus-minute-schedule-1,7c89c16eb6a83ccf,af56a138a40311a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:29.000 INFO 4486 --- [prometheus-minute-schedule-1,7c89c16eb6a83ccf,af56a138a40311a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:29.632 INFO 4486 --- [ocp-async-8,f895695c5bcd7b56,649b1f434d563161] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:30.103 INFO 4486 --- [prometheus-second-schedule-1,3c4809eaeae38562,0a318b0f679698ad] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:07:30.244 INFO 4486 --- [ocp-schedules-29,179d8a7153b3ff0a,af9e6975afcfd491] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 14:07:30.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f33659f05c714038,430982635d8d0e61] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f33659f05c714038, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:30.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,f33659f05c714038,430982635d8d0e61] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f33659f05c714038, duration=0 ms] 2024-10-16 14:07:30.984 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5a1f3070fa760018,5b6187414fee6d77] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=5a1f3070fa760018, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:07:30.985 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,5a1f3070fa760018,5b6187414fee6d77] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=5a1f3070fa760018, duration=2 ms] 2024-10-16 14:07:31.504 INFO 4486 --- [ocp-schedules-30,690e88571042b2f8,b41d40229a310888] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:07:31.632 INFO 4486 --- [ocp-async-4,babc9cf5c0c5ef20,ec37307c48a70725] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:31.648 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:31.994 INFO 4486 --- [prometheus-minute-schedule-1,420e37a2550ec6a2,9f37e40f9e666f42] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:31.999 INFO 4486 --- [prometheus-minute-schedule-1,420e37a2550ec6a2,9f37e40f9e666f42] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:32.188 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3fd0857621f154ce,3b0fd083b3f51599] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3fd0857621f154ce, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:32.188 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3fd0857621f154ce,3b0fd083b3f51599] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3fd0857621f154ce, duration=1 ms] 2024-10-16 14:07:32.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b362b9c14132e604,b208ab4bad075057] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b362b9c14132e604, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:32.474 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,b362b9c14132e604,b208ab4bad075057] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b362b9c14132e604, duration=1 ms] 2024-10-16 14:07:33.036 INFO 4486 --- [ocp-async-3,9c62c47507b7a204,df15629cd542d086] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:07:33.632 INFO 4486 --- [ocp-async-6,df9723423c354561,3c2c5e51fbd03b9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:34.650 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:34.994 INFO 4486 --- [prometheus-minute-schedule-1,3154b8c207cb28ac,883cd3ae63fe8d22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:35.000 INFO 4486 --- [prometheus-minute-schedule-1,3154b8c207cb28ac,883cd3ae63fe8d22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:35.632 INFO 4486 --- [ocp-async-3,b81b1ef37d42fdaa,c499b36d822faf22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:37.063 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:07:37.064 INFO 4486 --- [ocp-schedules-11,ba4d48b36eccd702,a13d34368227d0b2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:07:37.066 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:07:37.067 INFO 4486 --- [ocp-schedules-21,532facd8f6b02ef1,bc0355cf6f1d4e32] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:07:37.068 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:07:37.068 INFO 4486 --- [ocp-schedules-21,532facd8f6b02ef1,bc0355cf6f1d4e32] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:37.070 INFO 4486 --- [ocp-schedules-11,ba4d48b36eccd702,a13d34368227d0b2] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:07:37.081 INFO 4486 --- [ocp-schedules-1,f24cc3fe71fa6bef,a048df29b5b9fa05] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:07:37.083 INFO 4486 --- [ocp-schedules-1,f24cc3fe71fa6bef,a048df29b5b9fa05] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:07:37.089 INFO 4486 --- [ocp-schedules-29,1daebd250a923245,6cf307568fc2c90a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:07:37.090 INFO 4486 --- [host-check-clockdiff1,1daebd250a923245,9763f50a471802de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:07:37.090 INFO 4486 --- [ocp-schedules-9,b267529f2600a412,83a4736c506889d3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:07:37.090 INFO 4486 --- [host-check-clockdiff2,1daebd250a923245,5ae52c81a9b24ff3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:07:37.090 INFO 4486 --- [host-check-clockdiff3,1daebd250a923245,7e8cd17ecb4e6893] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:07:37.091 INFO 4486 --- [host-check-clockdiff4,1daebd250a923245,5a127eb0cbde77bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:07:37.091 INFO 4486 --- [ocp-schedules-17,95f3179e1adc9463,5c63a324008a7968] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:07:37.091 INFO 4486 --- [host-check-agent-status2,b267529f2600a412,0831c749bbb630d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:07:37.091 INFO 4486 --- [host-check-agent-status1,b267529f2600a412,da05b7bedd66515a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:07:37.092 INFO 4486 --- [host-check-agent-status3,b267529f2600a412,d274698870bbfd27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:07:37.092 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:07:37.092 INFO 4486 --- [host-check-agent-status4,b267529f2600a412,6b9f80c84bb5a4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:07:37.093 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:07:37.093 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:07:37.093 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:07:37.099 INFO 4486 --- [host-check-agent-status4,b267529f2600a412,6b9f80c84bb5a4d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:07:37.099 INFO 4486 --- [host-check-agent-status3,b267529f2600a412,d274698870bbfd27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:07:37.100 INFO 4486 --- [host-check-agent-status1,b267529f2600a412,da05b7bedd66515a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:37.100 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:07:37.106 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:07:37.107 INFO 4486 --- [host-check-agent-status3,b267529f2600a412,d274698870bbfd27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:07:37.107 INFO 4486 --- [host-check-agent-status1,b267529f2600a412,da05b7bedd66515a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:37.107 INFO 4486 --- [host-check-agent-status2,b267529f2600a412,0831c749bbb630d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:07:37.107 INFO 4486 --- [host-check-agent-status4,b267529f2600a412,6b9f80c84bb5a4d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:07:37.099 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:37.099 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:07:37.121 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:07:37.123 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:07:37.134 INFO 4486 --- [host-check-agent-status2,b267529f2600a412,0831c749bbb630d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:07:37.134 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:37.135 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:07:37.135 INFO 4486 --- [host-check-clockdiff1,1daebd250a923245,9763f50a471802de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058857 0 0 , error= 2024-10-16 14:07:37.140 INFO 4486 --- [host-check-clockdiff2,1daebd250a923245,5ae52c81a9b24ff3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058857 4 4 , error= 2024-10-16 14:07:37.143 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.144 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:37.145 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:07:37.162 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:07:37.162 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:07:37.164 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.165 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:37.166 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:07:37.166 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:07:37.165614241+08:00 2024-10-16 14:07:37.166 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:07:37.169730477+08:00 2024-10-16 14:07:37.167 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:07:37.167 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:07:37.166+08:00 2024-10-16 14:07:37.167 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:07:37.166+08:00 2024-10-16 14:07:37.167 INFO 4486 --- [host-check-agent-status2,b267529f2600a412,0831c749bbb630d8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:07:37.167 INFO 4486 --- [host-check-agent-status1,b267529f2600a412,da05b7bedd66515a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:07:37.167 INFO 4486 --- [host-check-agent-status3,b267529f2600a412,d274698870bbfd27] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:07:37.168 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:07:37.172116070+08:00 2024-10-16 14:07:37.169 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:07:37.166+08:00 and status AVAILABLE 2024-10-16 14:07:37.169 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:07:37.169+08:00 2024-10-16 14:07:37.169 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:07:37.170 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:07:37.166+08:00 and status AVAILABLE 2024-10-16 14:07:37.170 INFO 4486 --- [host-check-agent-status4,b267529f2600a412,6b9f80c84bb5a4d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:07:37.170 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:07:37.173127246+08:00 2024-10-16 14:07:37.170 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:07:37.170+08:00 2024-10-16 14:07:37.170 INFO 4486 --- [host-check-agent-status3,b267529f2600a412,d274698870bbfd27] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:07:37.170 INFO 4486 --- [host-check-agent-status3,b267529f2600a412,d274698870bbfd27] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:07:37.170 INFO 4486 --- [host-check-agent-status3,b267529f2600a412,d274698870bbfd27] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:07:37.172 INFO 4486 --- [host-check-agent-status1,b267529f2600a412,da05b7bedd66515a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:07:37.172 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:07:37.172 INFO 4486 --- [host-check-agent-status1,b267529f2600a412,da05b7bedd66515a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:07:37.172 INFO 4486 --- [host-check-agent-status1,b267529f2600a412,da05b7bedd66515a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:07:37.172 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:07:37.169+08:00 and status AVAILABLE 2024-10-16 14:07:37.172 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.172 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:07:37.173 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:07:37.170+08:00 and status AVAILABLE 2024-10-16 14:07:37.173 INFO 4486 --- [host-check-agent-status4,b267529f2600a412,6b9f80c84bb5a4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:07:37.173 INFO 4486 --- [host-check-agent-status2,b267529f2600a412,0831c749bbb630d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:07:37.173 INFO 4486 --- [host-check-agent-status2,b267529f2600a412,0831c749bbb630d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:07:37.173 INFO 4486 --- [host-check-agent-status4,b267529f2600a412,6b9f80c84bb5a4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:07:37.173 INFO 4486 --- [host-check-agent-status4,b267529f2600a412,6b9f80c84bb5a4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:07:37.173 INFO 4486 --- [host-check-agent-status2,b267529f2600a412,0831c749bbb630d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:07:37.174 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.174 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:07:37.175 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:07:37.175 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:07:37.176 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.176 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.177 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:07:37.177 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:07:37.182 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:37.183 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:07:37.184 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:37.184 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:07:37.185 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.185 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:07:37.186 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:07:37.186 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:07:37.186 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:07:37.187 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:07:37.189 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.189 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.189 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.189 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.189 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.190 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.195 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.196 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:07:37.205 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:37.205 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:37.205 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.209 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:07:37.211 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:37.211 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.213 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:07:37.213 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:07:37.213 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:07:37.223 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.224 INFO 4486 --- [ocp-schedules-12,5455784fde9d9266,b82f8ba3162bb6ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:07:37.238 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:07:37.243 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:07:37.245 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:07:37.251 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:07:37.256 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:07:37.256 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:07:37.263 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:07:37.268 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:37.271 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:37.271 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:37.272 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:07:37.288 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:07:37.300 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:07:37.301 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:07:37.310 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:07:37.310 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:07:37.311 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:07:37.320 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:07:37.326 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:37.328 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:37.338 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:37.349 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:07:37.354 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:07:37.356 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:07:37.359 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:07:37.363 INFO 4486 --- [host-check-host-status2,95f3179e1adc9463,959ed13672eb0a71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:07:37.367 INFO 4486 --- [host-check-host-status3,95f3179e1adc9463,009acf55129e8e8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:07:37.370 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:07:37.377 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:07:37.379 INFO 4486 --- [host-check-host-status4,95f3179e1adc9463,e9984d4c1a9df78e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:07:37.432 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:07:37.436 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:37.438 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.439 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:07:37.440 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.440 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.441 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:07:37.446 INFO 4486 --- [host-check-host-status1,95f3179e1adc9463,1e2b3efad63ae663] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:07:37.449 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.449 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.449 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.449 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.449 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.450 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.477 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:07:37.525 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:07:37.569 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:37.582 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:37.584 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.585 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:07:37.587 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:07:37.587 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.588 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:07:37.589 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.590 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.591 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:07:37.601 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.601 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.601 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.601 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.601 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.602 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.614 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:07:37.632 INFO 4486 --- [ocp-async-4,2733f8447ce1c3c6,70efe83b394a5a61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:37.651 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:37.802 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:07:37.847 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:37.849 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.849 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:07:37.851 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.852 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.852 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:07:37.862 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.862 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.862 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.862 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.862 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:37.863 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:37.880 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:07:37.964 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:07:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:07:37.994 INFO 4486 --- [prometheus-minute-schedule-1,770fc6ecec18a00c,87ada31c789a2826] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:38.000 INFO 4486 --- [prometheus-minute-schedule-1,770fc6ecec18a00c,87ada31c789a2826] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:38.063 INFO 4486 --- [host-check-clockdiff4,1daebd250a923245,5a127eb0cbde77bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058858 4 4 , error= 2024-10-16 14:07:38.097 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:07:38.141 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:38.143 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:38.144 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:07:38.145 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:38.146 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:38.147 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:07:38.156 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:38.156 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:38.156 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:38.156 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:38.156 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:07:38.156 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:38.220 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:07:38.422 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:07:38.223774, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:07:38.422 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:07:38.223774, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:07:38.462 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:07:38.223774} } 2024-10-16 14:07:38.483 INFO 4486 --- [ocp-schedules-12,0c0cd04b39e12ed5,9a8e70fb2c72ff9b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8187} 2024-10-16 14:07:38.484 INFO 4486 --- [ocp-schedules-12,0c0cd04b39e12ed5,9a8e70fb2c72ff9b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:07:38.525 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:07:38.535 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:07:38.538 INFO 4486 --- [ocp-schedules-18,9243032539083a13,169c825f96c1cee1] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:07:38.623 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,8be0cf0565964425,8773f9308508b9d4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8be0cf0565964425, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:38.623 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,8be0cf0565964425,8773f9308508b9d4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8be0cf0565964425, duration=2 ms] 2024-10-16 14:07:38.659 INFO 4486 --- [ocp-schedules-7,03e7684568b542a6,131a485237f05dc7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:07:38.685 WARN 4486 --- [scanner-13335413-1,03e7684568b542a6,5429e9c82c8f25d2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:07:38.894 INFO 4486 --- [host-check-clockdiff3,1daebd250a923245,7e8cd17ecb4e6893] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058858 4 4 , error= 2024-10-16 14:07:39.633 INFO 4486 --- [ocp-async-10,021692d93608301b,ce8e45491137e3aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:39.648 INFO 4486 --- [ocp-schedules-2,189c34f398a33d62,6e6db674edbee064] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:07:40.006 INFO 4486 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:07:40.007 INFO 4486 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:07:40.027 INFO 4486 --- [Check obproxy1,128c4bd60eac61e7,4fe2c970f2147b1f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:07:40.027 INFO 4486 --- [Check obproxy2,94fa6b8dea328b47,16e65b7ac0df54aa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:07:40.027 INFO 4486 --- [Check obproxy1,128c4bd60eac61e7,4fe2c970f2147b1f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:40.027 INFO 4486 --- [Check obproxy2,94fa6b8dea328b47,16e65b7ac0df54aa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:40.031 INFO 4486 --- [Collect obproxy2,236a3b615cf4b234,a6741a31379b1f98] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:07:40.032 INFO 4486 --- [Collect obproxy2,236a3b615cf4b234,a6741a31379b1f98] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:40.032 INFO 4486 --- [Collect obproxy1,111a5a35507b1ce5,dac7aa734d11b6c2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:07:40.032 INFO 4486 --- [Collect obproxy1,111a5a35507b1ce5,dac7aa734d11b6c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:07:40.050 INFO 4486 --- [Check obproxy2,94fa6b8dea328b47,16e65b7ac0df54aa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:07:40.050 INFO 4486 --- [Check obproxy2,94fa6b8dea328b47,16e65b7ac0df54aa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:07:40.052 INFO 4486 --- [Collect obproxy2,236a3b615cf4b234,a6741a31379b1f98] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:07:40.055 INFO 4486 --- [Check obproxy1,128c4bd60eac61e7,4fe2c970f2147b1f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:07:40.055 INFO 4486 --- [Check obproxy1,128c4bd60eac61e7,4fe2c970f2147b1f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:07:40.059 INFO 4486 --- [Collect obproxy2,236a3b615cf4b234,a6741a31379b1f98] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:07:40.063 INFO 4486 --- [Collect obproxy1,111a5a35507b1ce5,dac7aa734d11b6c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:07:40.064 INFO 4486 --- [Collect obproxy2,236a3b615cf4b234,a6741a31379b1f98] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:07:40.069 INFO 4486 --- [Collect obproxy1,111a5a35507b1ce5,dac7aa734d11b6c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:07:40.073 INFO 4486 --- [Collect obproxy1,111a5a35507b1ce5,dac7aa734d11b6c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:07:40.635 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2ed721f1b1ccd2a6,6f9686699c7cd117] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2ed721f1b1ccd2a6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:40.636 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2ed721f1b1ccd2a6,6f9686699c7cd117] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2ed721f1b1ccd2a6, duration=1 ms] 2024-10-16 14:07:40.652 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:40.994 INFO 4486 --- [prometheus-minute-schedule-1,ef1a377285e7f0be,2f7e916a7e2864d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:40.995 INFO 4486 --- [prometheus-hour-schedule-1,abef0ba5d5d75b09,4cb358621d339508] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:07:41.000 INFO 4486 --- [prometheus-minute-schedule-1,ef1a377285e7f0be,2f7e916a7e2864d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:41.510 INFO 4486 --- [ocp-schedules-8,acb38dc6d20728b4,8756686b9a74d69a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:07:41.655 INFO 4486 --- [ocp-async-8,61b59d6b4db7edfe,be984508e3014302] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:42.193 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c19e48f6484d9328,2cb94b23035f4e23] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c19e48f6484d9328, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:42.193 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c19e48f6484d9328,2cb94b23035f4e23] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c19e48f6484d9328, duration=1 ms] 2024-10-16 14:07:42.205 INFO 4486 --- [ocp-schedules-20,4a120efc1f029bb2,ea1736e64191d73e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:07:42.479 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d165c73db337f069,7e2b5834944091fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d165c73db337f069, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:42.479 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,d165c73db337f069,7e2b5834944091fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d165c73db337f069, duration=1 ms] 2024-10-16 14:07:43.036 INFO 4486 --- [ocp-async-7,a3e702c593c3483c,8b542cb5c0ce11f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:07:43.653 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:43.655 INFO 4486 --- [ocp-async-4,849522df25e2420f,82c877c932fdb0cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:43.994 INFO 4486 --- [prometheus-minute-schedule-1,c062d17108e1565f,67587bb0d226159e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:44.001 INFO 4486 --- [prometheus-minute-schedule-1,c062d17108e1565f,67587bb0d226159e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:45.630 INFO 4486 --- [ocp-schedules-25,efb8dbfe3d4dd34c,89ceb467307a4c71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:07:45.641 INFO 4486 --- [ocp-schedules-25,efb8dbfe3d4dd34c,89ceb467307a4c71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:07:45.641 INFO 4486 --- [ocp-schedules-25,efb8dbfe3d4dd34c,89ceb467307a4c71] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:07:45.655 INFO 4486 --- [ocp-async-10,3eb36bb5b7566778,2c992852293b3a14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:45.664 INFO 4486 --- [ocp-schedules-7,57eeed353c2d7a67,e7f7624e61defdc0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:07:45.664 INFO 4486 --- [ocp-schedules-7,57eeed353c2d7a67,e7f7624e61defdc0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:07:46.653 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:46.994 INFO 4486 --- [prometheus-minute-schedule-1,1d7d085830ad222b,6e578d3953ccf839] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:47.000 INFO 4486 --- [prometheus-minute-schedule-1,1d7d085830ad222b,6e578d3953ccf839] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:47.070 INFO 4486 --- [ocp-schedules-21,532facd8f6b02ef1,bc0355cf6f1d4e32] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:07:47.171 INFO 4486 --- [ocp-schedules-1,f24cc3fe71fa6bef,a048df29b5b9fa05] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:07:47.321 INFO 4486 --- [ocp-schedules-19,f57f51b5040c41d5,e3a2df2b44534a21] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:07:47.655 INFO 4486 --- [ocp-async-2,cf18f82626d5fa5b,601938dc216c993b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:48.130 INFO 4486 --- [ocp-schedules-16,25e061e7672c1a7c,86646b85fecc2704] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:07:48.130 INFO 4486 --- [ocp-schedules-16,37d6b4abecc27137,506127ceb669e1db] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:07:48.132 INFO 4486 --- [ocp-schedules-16,37d6b4abecc27137,506127ceb669e1db] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:48.160 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.160 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:48.170 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.171 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:48.196 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.197 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.197 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:07:48.203 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.204 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:48.224 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.225 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.225 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:48.226 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:07:48.226 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.229 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:07:48.230 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:48.238 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.238 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:48.249 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.258 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.259 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.259 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:48.260 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:07:48.265 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:07:48.270 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:48.272 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:48.295 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:07:48.312 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.314 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.315 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:07:48.328 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:07:48.328 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:07:48.368 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.369 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.370 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:07:48.373 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.377 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:07:48.378 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:07:48.380 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:48.391 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.392 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.392 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:07:48.401 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:07:48.401 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058286918000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:07:48.409 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.409 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:48.439 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.439 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:48.468 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.468 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:48.501 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.502 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:07:48.529 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.529 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:07:48.556 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:07:48.585 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.586 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.587 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:07:48.591 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:07:48.596 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:07:48.603 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:07:48.605 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:07:48.625 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:07:48.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,269aeaed9e0fe2db,341cf44a2df3ed2d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=269aeaed9e0fe2db, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:48.631 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,269aeaed9e0fe2db,341cf44a2df3ed2d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=269aeaed9e0fe2db, duration=2 ms] 2024-10-16 14:07:48.752 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:07:48.758 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 14:07:48.781 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:07:48.783 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:07:48.784 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:07:48.795 INFO 4486 --- [ocp-schedules-29,914981a04019ddfb,803473b2849c8906] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:07:48.818 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:07:48.818 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:07:48.829 WARN 4486 --- [scanner-13335656-1,914981a04019ddfb,1177077cf77e9b02] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:07:48.934 INFO 4486 --- [alarm-detect-15,914981a04019ddfb,e9be2e262766915e] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058835 ~ 1729058858 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058850, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058850, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058850=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:07:49.654 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:49.655 INFO 4486 --- [ocp-async-4,d24fccee15c42298,d528bf5e6244a6fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:49.994 INFO 4486 --- [prometheus-minute-schedule-1,a2219f6efc7ef763,9e29131a6a3ffc93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:50.001 INFO 4486 --- [prometheus-minute-schedule-1,a2219f6efc7ef763,9e29131a6a3ffc93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:50.608 INFO 4486 --- [ocp-schedules-6,9cb851e53fb9d97d,8e4a2e9d06ebd25b] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:07:50.608 INFO 4486 --- [ocp-schedules-6,9cb851e53fb9d97d,8e4a2e9d06ebd25b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:07:50.609 INFO 4486 --- [ocp-schedules-6,9cb851e53fb9d97d,8e4a2e9d06ebd25b] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:07:50.609 INFO 4486 --- [ocp-schedules-6,9cb851e53fb9d97d,8e4a2e9d06ebd25b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:07:50.610 INFO 4486 --- [ocp-schedules-6,9cb851e53fb9d97d,8e4a2e9d06ebd25b] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:07:50.610 INFO 4486 --- [ocp-schedules-6,9cb851e53fb9d97d,8e4a2e9d06ebd25b] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703046246634950656 2024-10-16 14:07:50.643 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f545408aced5f25d,a56d6c5e48adbbce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=f545408aced5f25d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:50.644 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,f545408aced5f25d,a56d6c5e48adbbce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=f545408aced5f25d, duration=2 ms] 2024-10-16 14:07:51.515 INFO 4486 --- [ocp-schedules-27,2ccff3b8288d770c,2652375e63b36696] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:07:51.656 INFO 4486 --- [ocp-async-7,1766506079c94ddc,82d39777cedb4bde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:51.835 INFO 4486 --- [ocp-schedules-14,ad7c82abf3cedac7,5f08b5d613025949] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:07:51.835 INFO 4486 --- [ocp-schedules-14,ad7c82abf3cedac7,5f08b5d613025949] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:07:51.835 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:07:51.835 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:07:51.835 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:07:51.835 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:07:51.835 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:07:51.840 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:07:51.840 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:07:51.840 INFO 4486 --- [ocp-schedules-14,ad7c82abf3cedac7,5f08b5d613025949] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:07:51.840 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:07:51.841 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:07:51.841 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:07:51.841 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:07:51.842 INFO 4486 --- [ocp-schedules-14,ad7c82abf3cedac7,5f08b5d613025949] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:07:51.842 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:07:51.842 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:07:51.842 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:07:51.842 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:07:51.863 INFO 4486 --- [ocp-schedules-14,ad7c82abf3cedac7,5f08b5d613025949] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:07:51.864 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:07:51.865 INFO 4486 --- [ForkJoinPool.commonPool-worker-25,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:07:51.866 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:07:51.868 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:07:51.869 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:07:51.871 INFO 4486 --- [ocp-schedules-14,ad7c82abf3cedac7,5f08b5d613025949] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:07:52.200 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,88cf6aac73a0ab0b,6ca79c56b60d2078] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=88cf6aac73a0ab0b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:52.200 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,88cf6aac73a0ab0b,6ca79c56b60d2078] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=88cf6aac73a0ab0b, duration=1 ms] 2024-10-16 14:07:52.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,0e4f0b875a78efd6,6fe64a77e9296546] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0e4f0b875a78efd6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:52.484 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,0e4f0b875a78efd6,6fe64a77e9296546] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0e4f0b875a78efd6, duration=2 ms] 2024-10-16 14:07:52.538 INFO 4486 --- [ocp-schedules-1,096b7cba1c803b26,dd913fdc58d3fc57] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:07:52.655 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:52.994 INFO 4486 --- [prometheus-minute-schedule-1,6431b6ff9bd35102,9d190928d42637ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:53.000 INFO 4486 --- [prometheus-minute-schedule-1,6431b6ff9bd35102,9d190928d42637ba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:53.037 INFO 4486 --- [ocp-async-1,1e586ad257afc9f3,904d446d1595ed6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:07:53.656 INFO 4486 --- [ocp-async-2,e57b6cb6f592627c,9166f6998ed9c6db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:55.656 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:55.656 INFO 4486 --- [ocp-async-5,d24c41add9e49149,1a3c0f7184165f66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:55.994 INFO 4486 --- [prometheus-minute-schedule-1,4fed88417c5bb643,eeb8573a0f9cfcf7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:55.999 INFO 4486 --- [prometheus-minute-schedule-1,4fed88417c5bb643,eeb8573a0f9cfcf7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:57.299 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a9d1550fb9ac1fd5,1d9990a1d4259784] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=a9d1550fb9ac1fd5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:07:57.300 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,a9d1550fb9ac1fd5,1d9990a1d4259784] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=a9d1550fb9ac1fd5, duration=2 ms] 2024-10-16 14:07:57.656 INFO 4486 --- [ocp-async-10,fe481059fca46c50,c905c19c13fc6853] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:07:58.635 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,478915b514993ae5,cc34d5360b4cf50d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=478915b514993ae5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:07:58.635 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,478915b514993ae5,cc34d5360b4cf50d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=478915b514993ae5, duration=1 ms] 2024-10-16 14:07:58.657 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:07:58.994 INFO 4486 --- [prometheus-minute-schedule-1,38eb55a3ce5931fb,ae8a7c108330995c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058880, interval=60 2024-10-16 14:07:58.999 INFO 4486 --- [prometheus-minute-schedule-1,38eb55a3ce5931fb,ae8a7c108330995c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:07:59.032 INFO 4486 --- [ocp-schedules-26,6501a3c908887be5,ac7f14e179b51532] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:07:59.063 WARN 4486 --- [scanner-13335977-1,6501a3c908887be5,b69be1e92be5ac96] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:07:59.108 INFO 4486 --- [ocp-schedules-19,85614e2e81905266,7b905c3b0678396e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 14:07:59.117 INFO 4486 --- [ocp-schedules-19,85614e2e81905266,7b905c3b0678396e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:07:59.117 INFO 4486 --- [alarm-detect-14,6501a3c908887be5,b94c38e1fc58f2e1] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058850 ~ 1729058869 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058865, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058865, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058865=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:07:59.655 INFO 4486 --- [ocp-schedules-9,7c3fdd6f9d0af93b,6df9a5010f1cfb48] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:07:59.656 INFO 4486 --- [ocp-async-6,4dda7a4af27acaf8,ad8f1a90373a771d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:00.652 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,2a69f9fc8756e5d4,b113e90cf2cc30f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2a69f9fc8756e5d4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:00.652 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,2a69f9fc8756e5d4,b113e90cf2cc30f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2a69f9fc8756e5d4, duration=1 ms] 2024-10-16 14:08:00.954 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2024-10-16 14:08:00.986 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:00.993 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:01.019 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:01.021 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:01.047 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2024-10-16 14:08:01.048 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2024-10-16 14:08:01.071 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2024-10-16 14:08:01.073 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2024-10-16 14:08:01.098 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:01.100 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:01.128 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:08:01.130 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:08:01.152 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:08:01.153 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:08:01.177 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000002 2024-10-16 14:08:01.179 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000002 2024-10-16 14:08:01.204 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:08:01.206 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:08:01.229 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000016 2024-10-16 14:08:01.230 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000016 2024-10-16 14:08:01.254 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:01.256 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:01.277 INFO 4486 --- [ocp-schedules-13,a79664ada874f87b,4bb12268ca56daf9] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=11, cachedSize=11 2024-10-16 14:08:01.522 INFO 4486 --- [ocp-schedules-11,60779faf23d27d2d,adf885a1b887d7ad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:08:01.657 INFO 4486 --- [ocp-async-1,f4339830e4383667,7d255254a341b02f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:01.657 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:01.994 INFO 4486 --- [prometheus-minute-schedule-1,f55a2ab881d32a47,f2e54e7d9f2b2f86] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:02.000 INFO 4486 --- [prometheus-minute-schedule-1,f55a2ab881d32a47,f2e54e7d9f2b2f86] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:02.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,55b408e6f0ce38c9,55f49df9ebcd80c7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=55b408e6f0ce38c9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:02.205 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,55b408e6f0ce38c9,55f49df9ebcd80c7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=55b408e6f0ce38c9, duration=1 ms] 2024-10-16 14:08:02.488 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6952efb7f6a11141,03ff010b7fd90ab6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6952efb7f6a11141, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:02.489 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,6952efb7f6a11141,03ff010b7fd90ab6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6952efb7f6a11141, duration=2 ms] 2024-10-16 14:08:02.726 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,48862f4e52f020be,f66e7d570aa68700] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=48862f4e52f020be, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:08:02.726 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,48862f4e52f020be,f66e7d570aa68700] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=48862f4e52f020be, duration=1 ms] 2024-10-16 14:08:03.037 INFO 4486 --- [ocp-async-10,a70ef2e9bf25dc58,09146db4a8c3cb45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:08:03.069 INFO 4486 --- [ocp-schedules-30,f0f2d91fece4b3ce,f31ac511f0731672] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:08:03.069+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:08:03.073 INFO 4486 --- [ocp-schedules-30,f0f2d91fece4b3ce,f31ac511f0731672] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:08:03.083 INFO 4486 --- [ocp-schedules-30,f0f2d91fece4b3ce,f31ac511f0731672] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:08:03.094 INFO 4486 --- [ocp-schedules-30,f0f2d91fece4b3ce,f31ac511f0731672] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:08:03.657 INFO 4486 --- [ocp-async-9,16a65eb066319388,2fdcb9877e23dbcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:04.658 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:04.994 INFO 4486 --- [prometheus-minute-schedule-1,fd57ed253696e3a0,a5336d63fd133e0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:05.001 INFO 4486 --- [prometheus-minute-schedule-1,fd57ed253696e3a0,a5336d63fd133e0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:05.657 INFO 4486 --- [ocp-async-10,a59e29a750c477a2,8475d0202282a8bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:06.918 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,6deaf62068198076,c1605c9738dd7ea7] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=OceanBase&version=2&ObClusterId=1701313637, client=10.197.49.184, traceId=6deaf62068198076, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=OceanBase,1701313637,2,{"ObRegion":"OceanBase","ObCluster":"OceanBase","ObRegionId":1701313637,"ObClusterId":1701313637,"Type":"PRIMARY","timestamp":1729058886911834,"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 14:08:06.932 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,6deaf62068198076,c1605c9738dd7ea7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=OceanBase, obClusterId=1701313637, rootServerJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058886919000}, groupName=OceanBase, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 14:08:06.932 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,6deaf62068198076,c1605c9738dd7ea7] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 14:08:06.933 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,6deaf62068198076,c1605c9738dd7ea7] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=OceanBase, clusterId=1701313637, affectRows=1 2024-10-16 14:08:06.933 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,6deaf62068198076,c1605c9738dd7ea7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.184, traceId=6deaf62068198076, duration=20 ms] 2024-10-16 14:08:07.066 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:08:07.067 INFO 4486 --- [ocp-schedules-5,e906bebe52a53f58,696c81bfa201e892] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:08:07.069 INFO 4486 --- [ocp-schedules-5,e906bebe52a53f58,696c81bfa201e892] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:07.081 INFO 4486 --- [ocp-schedules-28,9d68c7d8d3d70dde,3abac4e4778926f2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:08:07.084 INFO 4486 --- [ocp-schedules-28,9d68c7d8d3d70dde,3abac4e4778926f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:07.086 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:08:07.109 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.110 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:07.111 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:08:07.119 INFO 4486 --- [ocp-schedules-23,54123e4183e5f918,fe8f164be15a70de] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:08:07.119 INFO 4486 --- [ocp-schedules-29,6bb48afaaf785381,f24211d287530b6f] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:08:07.120 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:08:07.120 INFO 4486 --- [host-check-agent-status1,54123e4183e5f918,d00199b954b686f2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:08:07.121 INFO 4486 --- [host-check-agent-status2,54123e4183e5f918,8578239c81b86999] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:08:07.122 INFO 4486 --- [host-check-agent-status3,54123e4183e5f918,9caa7a483ff65529] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:08:07.122 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:08:07.122 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:08:07.123 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:08:07.123 INFO 4486 --- [host-check-agent-status4,54123e4183e5f918,ea934538b9959d2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:08:07.126 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.127 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:07.132 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:08:07.154 INFO 4486 --- [ocp-schedules-12,d9b79014387b8e15,95b3ab18e7e59719] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:08:07.155 INFO 4486 --- [host-check-clockdiff1,d9b79014387b8e15,ac897eabe210e234] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:08:07.155 INFO 4486 --- [host-check-clockdiff2,d9b79014387b8e15,395e7e8ba906fc42] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:08:07.155 INFO 4486 --- [host-check-clockdiff3,d9b79014387b8e15,d102ef474fcbcecc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:08:07.155 INFO 4486 --- [host-check-clockdiff4,d9b79014387b8e15,b163c5d7949f1b41] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:08:07.156 INFO 4486 --- [host-check-agent-status4,54123e4183e5f918,ea934538b9959d2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:07.157 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:07.158 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:07.158 INFO 4486 --- [host-check-agent-status2,54123e4183e5f918,8578239c81b86999] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:07.159 INFO 4486 --- [host-check-agent-status4,54123e4183e5f918,ea934538b9959d2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:07.159 INFO 4486 --- [host-check-agent-status3,54123e4183e5f918,9caa7a483ff65529] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:07.159 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:07.159 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:07.160 INFO 4486 --- [host-check-agent-status2,54123e4183e5f918,8578239c81b86999] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:07.160 INFO 4486 --- [host-check-agent-status3,54123e4183e5f918,9caa7a483ff65529] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:07.161 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:07.165 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:07.210 INFO 4486 --- [host-check-clockdiff4,d9b79014387b8e15,b163c5d7949f1b41] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058887 4 4 , error= 2024-10-16 14:08:07.223 INFO 4486 --- [host-check-clockdiff1,d9b79014387b8e15,ac897eabe210e234] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058887 0 0 , error= 2024-10-16 14:08:07.223 INFO 4486 --- [host-check-clockdiff2,d9b79014387b8e15,395e7e8ba906fc42] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058887 4 4 , error= 2024-10-16 14:08:07.225 INFO 4486 --- [host-check-agent-status1,54123e4183e5f918,d00199b954b686f2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:07.230 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:07.230 INFO 4486 --- [host-check-agent-status1,54123e4183e5f918,d00199b954b686f2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:07.231 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.232 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:07.236 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:08:07.246 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.247 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:07.247 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:08:07.249 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:08:07.251 INFO 4486 --- [host-check-agent-status2,54123e4183e5f918,8578239c81b86999] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:08:07.251 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:08:07.252 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:08:07.254879490+08:00 2024-10-16 14:08:07.252 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:08:07.255113625+08:00 2024-10-16 14:08:07.252 INFO 4486 --- [host-check-agent-status4,54123e4183e5f918,ea934538b9959d2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:08:07.252 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:08:07.252+08:00 2024-10-16 14:08:07.252 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:08:07.252+08:00 2024-10-16 14:08:07.253 INFO 4486 --- [host-check-agent-status3,54123e4183e5f918,9caa7a483ff65529] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:08:07.254 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:08:07.258017164+08:00 2024-10-16 14:08:07.255 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:08:07.254+08:00 2024-10-16 14:08:07.255 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:08:07.252+08:00 and status AVAILABLE 2024-10-16 14:08:07.255 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:08:07.252+08:00 and status AVAILABLE 2024-10-16 14:08:07.256 INFO 4486 --- [host-check-agent-status2,54123e4183e5f918,8578239c81b86999] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:08:07.256 INFO 4486 --- [host-check-agent-status4,54123e4183e5f918,ea934538b9959d2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:08:07.256 INFO 4486 --- [host-check-agent-status2,54123e4183e5f918,8578239c81b86999] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:08:07.256 INFO 4486 --- [host-check-agent-status4,54123e4183e5f918,ea934538b9959d2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:08:07.256 INFO 4486 --- [host-check-agent-status2,54123e4183e5f918,8578239c81b86999] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:08:07.256 INFO 4486 --- [host-check-agent-status4,54123e4183e5f918,ea934538b9959d2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:08:07.257 INFO 4486 --- [host-check-agent-status1,54123e4183e5f918,d00199b954b686f2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:08:07.258 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:08:07.254+08:00 and status AVAILABLE 2024-10-16 14:08:07.262 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:08:07.262 INFO 4486 --- [host-check-agent-status3,54123e4183e5f918,9caa7a483ff65529] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:08:07.262 INFO 4486 --- [host-check-agent-status3,54123e4183e5f918,9caa7a483ff65529] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:08:07.262 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:08:07.262 INFO 4486 --- [host-check-agent-status3,54123e4183e5f918,9caa7a483ff65529] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:08:07.264 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:08:07.265 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:08:07.266 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:08:07.266221917+08:00 2024-10-16 14:08:07.267 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.267 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:08:07.266+08:00 2024-10-16 14:08:07.267 INFO 4486 --- [host-check-agent-status1,54123e4183e5f918,d00199b954b686f2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:08:07.267 INFO 4486 --- [host-check-agent-status1,54123e4183e5f918,d00199b954b686f2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:08:07.267 INFO 4486 --- [host-check-agent-status1,54123e4183e5f918,d00199b954b686f2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:08:07.268 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:07.271 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:08:07.266+08:00 and status AVAILABLE 2024-10-16 14:08:07.273 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:08:07.280 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.281 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:07.284 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.285 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:08:07.295 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:07.296 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:07.297 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:07.297 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:07.299 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:07.299 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:07.299 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:07.300 INFO 4486 --- [ocp-schedules-25,ecd37f6a54ad03a2,1ca83ee89edca243] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:08:07.302 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:07.304 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:07.320 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:07.321 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:07.321 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:07.329 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:07.354 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:08:07.354 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:08:07.355 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:08:07.374 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:08:07.375 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:08:07.375 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:08:07.381 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:08:07.395 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:08:07.405 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:07.406 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:07.406 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:07.411 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:07.443 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:08:07.446 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:08:07.455 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:08:07.457 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:08:07.457 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:08:07.468 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:08:07.469 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:08:07.472 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:07.474 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:07.479 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:08:07.485 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:07.494 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:07.499 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:08:07.508 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:08:07.511 INFO 4486 --- [host-check-host-status3,6bb48afaaf785381,347e957e87bb7a22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:08:07.519 INFO 4486 --- [host-check-host-status4,6bb48afaaf785381,4b2d25050a07cd5b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:08:07.522 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:08:07.604 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:08:07.604 INFO 4486 --- [host-check-host-status2,6bb48afaaf785381,a57434f44f31cacc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:08:07.615 INFO 4486 --- [host-check-host-status1,6bb48afaaf785381,1504e772d01b21e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:08:07.658 INFO 4486 --- [ocp-async-2,d89dc6b153e18ef5,c73dc1c4f2ce870d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:07.659 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:07.994 INFO 4486 --- [prometheus-minute-schedule-1,75ac17945c7567a0,70ef9bf4b291d166] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:08.001 INFO 4486 --- [prometheus-minute-schedule-1,75ac17945c7567a0,70ef9bf4b291d166] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:08.641 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,94964de27d29837b,90fff0b2202e7a41] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=94964de27d29837b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:08.641 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,94964de27d29837b,90fff0b2202e7a41] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=94964de27d29837b, duration=1 ms] 2024-10-16 14:08:08.962 INFO 4486 --- [host-check-clockdiff3,d9b79014387b8e15,d102ef474fcbcecc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058888 4 4 , error= 2024-10-16 14:08:09.157 INFO 4486 --- [ocp-schedules-19,85614e2e81905266,7b905c3b0678396e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 14:08:09.178 INFO 4486 --- [ocp-schedules-24,7b3c3cd02a99040c,9097cdba4bb1cc44] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:08:09.224 WARN 4486 --- [scanner-13336229-1,7b3c3cd02a99040c,30dc6b355c2f05e5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:08:09.658 INFO 4486 --- [ocp-async-3,bcc2cb09b36a56cd,cb30b83b069d7f50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:10.018 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:08:10.018 INFO 4486 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:08:10.052 INFO 4486 --- [Collect obproxy2,3546a2de6dff8dad,8a873d318eadf21e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:08:10.052 INFO 4486 --- [Check obproxy1,c030d878953a1db8,17624b0386fade13] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:08:10.052 INFO 4486 --- [Check obproxy2,1860e6661a31237a,e178b82671f644b2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:08:10.052 INFO 4486 --- [Collect obproxy2,3546a2de6dff8dad,8a873d318eadf21e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:10.052 INFO 4486 --- [Check obproxy1,c030d878953a1db8,17624b0386fade13] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:10.052 INFO 4486 --- [Check obproxy2,1860e6661a31237a,e178b82671f644b2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:10.057 INFO 4486 --- [Collect obproxy1,a5f0affd15fe5b6b,583e66f22937de73] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:08:10.057 INFO 4486 --- [Collect obproxy1,a5f0affd15fe5b6b,583e66f22937de73] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:10.073 INFO 4486 --- [Collect obproxy2,3546a2de6dff8dad,8a873d318eadf21e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:08:10.076 INFO 4486 --- [Check obproxy2,1860e6661a31237a,e178b82671f644b2] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:08:10.076 INFO 4486 --- [Check obproxy2,1860e6661a31237a,e178b82671f644b2] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:08:10.078 INFO 4486 --- [Collect obproxy2,3546a2de6dff8dad,8a873d318eadf21e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:08:10.082 INFO 4486 --- [Check obproxy1,c030d878953a1db8,17624b0386fade13] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:08:10.082 INFO 4486 --- [Check obproxy1,c030d878953a1db8,17624b0386fade13] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:08:10.084 INFO 4486 --- [Collect obproxy2,3546a2de6dff8dad,8a873d318eadf21e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:08:10.089 INFO 4486 --- [Collect obproxy1,a5f0affd15fe5b6b,583e66f22937de73] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:08:10.092 INFO 4486 --- [Collect obproxy1,a5f0affd15fe5b6b,583e66f22937de73] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:08:10.095 INFO 4486 --- [Collect obproxy1,a5f0affd15fe5b6b,583e66f22937de73] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:08:10.656 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e211be98515cd32f,ea3f39e7953f2684] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e211be98515cd32f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:10.656 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,e211be98515cd32f,ea3f39e7953f2684] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e211be98515cd32f, duration=1 ms] 2024-10-16 14:08:10.660 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:10.975 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:08:10.994 INFO 4486 --- [prometheus-minute-schedule-1,3162745ac1f6851b,bde3e2481b82e7f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:11.001 INFO 4486 --- [prometheus-minute-schedule-1,3162745ac1f6851b,bde3e2481b82e7f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:11.005 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:08:11.527 INFO 4486 --- [ocp-schedules-7,90046b97adf8919d,18cadcf36019b19a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:08:11.658 INFO 4486 --- [ocp-async-6,3bdf4104c847ca9f,15cc5417eea878b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:11.851 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=2, affectRows=2 2024-10-16 14:08:11.851 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=2, affectRows=2 2024-10-16 14:08:11.851 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=2, affectRows=2 2024-10-16 14:08:11.850 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=2, affectRows=2 2024-10-16 14:08:11.861 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=1, affectRows=1 2024-10-16 14:08:11.861 INFO 4486 --- [metric-archive-second-30,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=1, validCount=1 2024-10-16 14:08:11.866 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=2, validCount=2 2024-10-16 14:08:11.866 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=2, validCount=2 2024-10-16 14:08:11.867 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=2, validCount=2 2024-10-16 14:08:11.867 INFO 4486 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=2, validCount=2 2024-10-16 14:08:11.851 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=2, affectRows=2 2024-10-16 14:08:11.868 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=2, validCount=2 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=1, affectRows=1 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=95, affectRows=95 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=1, validCount=1 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=47, affectRows=47 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=1, affectRows=1 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=1, validCount=1 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=47, validCount=47 2024-10-16 14:08:11.869 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=95, validCount=95 2024-10-16 14:08:11.871 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=2, affectRows=2 2024-10-16 14:08:11.871 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=2, validCount=2 2024-10-16 14:08:11.877 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=130, affectRows=130 2024-10-16 14:08:11.882 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=190, affectRows=190 2024-10-16 14:08:11.882 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=1000, affectRows=1000 2024-10-16 14:08:11.882 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=208, affectRows=208 2024-10-16 14:08:11.883 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=206, affectRows=206 2024-10-16 14:08:11.884 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=352, affectRows=352 2024-10-16 14:08:11.885 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=206, affectRows=206 2024-10-16 14:08:11.891 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=130, validCount=130 2024-10-16 14:08:11.893 INFO 4486 --- [metric-archive-second-12,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=206, validCount=206 2024-10-16 14:08:11.893 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=206, validCount=206 2024-10-16 14:08:11.893 INFO 4486 --- [metric-archive-second-22,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=190, validCount=190 2024-10-16 14:08:11.894 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=352, validCount=352 2024-10-16 14:08:11.894 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=206, validCount=206 2024-10-16 14:08:11.896 INFO 4486 --- [metric-archive-second-17,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=999, validCount=999 2024-10-16 14:08:11.941 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058820, nodeCount=1000, affectRows=1000 2024-10-16 14:08:11.952 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058820, nodeCount=1000, validCount=1000 2024-10-16 14:08:12.209 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,76ffeadf8eccaad3,9a67d0ecae295415] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=76ffeadf8eccaad3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:12.210 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,76ffeadf8eccaad3,9a67d0ecae295415] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=76ffeadf8eccaad3, duration=1 ms] 2024-10-16 14:08:12.492 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,af1fdc5734377acb,c7f74e157e9342ad] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=af1fdc5734377acb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:12.493 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,af1fdc5734377acb,c7f74e157e9342ad] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=af1fdc5734377acb, duration=1 ms] 2024-10-16 14:08:13.038 INFO 4486 --- [ocp-async-5,a69c1ca91e9619f1,509df5e1eaea185e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:08:13.598 INFO 4486 --- [ocp-schedules-7,be97fb0199a0869a,a4af9d4f368a65b7] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:08:13.600 INFO 4486 --- [ocp-schedules-7,be97fb0199a0869a,a4af9d4f368a65b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:13.601 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:08:13.601 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:08:13.601 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:08:13.658 INFO 4486 --- [ocp-async-2,f86edfe23a2eeaab,32305a3d74dc68a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:13.661 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:14.038 INFO 4486 --- [prometheus-minute-schedule-1,5ed106e2d0e26726,0b4eeeda5be28b8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:14.044 INFO 4486 --- [prometheus-minute-schedule-1,5ed106e2d0e26726,0b4eeeda5be28b8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:15.416 INFO 4486 --- [metric-archive-async-31,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058820, nodeCount=477, affectRows=477 2024-10-16 14:08:15.658 INFO 4486 --- [ocp-async-5,9da190c8d52473aa,1b3b69038c20d82e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:16.044 INFO 4486 --- [metric-archive-async-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058820, nodeCount=477, validCount=477 2024-10-16 14:08:16.662 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:16.994 INFO 4486 --- [prometheus-minute-schedule-1,28ed4af88bee255f,69ad061d4543cd01] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:16.999 INFO 4486 --- [prometheus-minute-schedule-1,28ed4af88bee255f,69ad061d4543cd01] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:17.071 INFO 4486 --- [ocp-schedules-5,e906bebe52a53f58,696c81bfa201e892] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:08:17.133 INFO 4486 --- [ocp-schedules-28,9d68c7d8d3d70dde,3abac4e4778926f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:08:17.324 INFO 4486 --- [ocp-schedules-27,b02d8b5464924454,942380ff997ca442] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:08:17.659 INFO 4486 --- [ocp-async-6,056719967decdd4a,94dd38b147605580] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:18.645 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,5a8e376beb9c1560,85c84794ab4ae0ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5a8e376beb9c1560, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:18.645 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,5a8e376beb9c1560,85c84794ab4ae0ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5a8e376beb9c1560, duration=1 ms] 2024-10-16 14:08:18.819 INFO 4486 --- [ocp-schedules-16,37d6b4abecc27137,506127ceb669e1db] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:08:18.820 INFO 4486 --- [ocp-schedules-16,9ae032ef7ef3f0b3,2cc5ccd40a439c59] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:08:18.828 INFO 4486 --- [ocp-schedules-16,9ae032ef7ef3f0b3,2cc5ccd40a439c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:18.861 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:18.861 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:18.873 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:18.874 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:18.899 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:18.900 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:18.956 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:18.957 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:18.958 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:08:18.995 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:18.996 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:18.997 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:18.998 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:08:18.998 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.001 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:08:19.003 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:19.017 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.018 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:19.036 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.046 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:19.047 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.047 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:19.048 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:08:19.054 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:19.054 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:08:19.055 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.056 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:08:19.060 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:19.064 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:19.090 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:08:19.090 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:08:19.091 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:08:19.112 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:19.113 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.114 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:19.118 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.121 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:08:19.124 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:19.152 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.152 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:19.173 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:08:19.181 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.181 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:19.183 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:19.184 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.185 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:08:19.194 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:08:19.194 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058886919000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:08:19.209 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.209 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:19.236 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.237 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:19.262 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.262 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:19.291 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:19.318 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:19.320 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.321 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:19.325 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:08:19.331 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:08:19.336 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:08:19.338 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:08:19.363 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:08:19.383 INFO 4486 --- [ocp-schedules-3,6e7967f1e0c729d8,6b1861c1a0e322dc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:08:19.407 WARN 4486 --- [scanner-13337370-1,6e7967f1e0c729d8,0478e0b9b120f884] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:08:19.501 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:08:19.502 INFO 4486 --- [alarm-detect-7,6e7967f1e0c729d8,7c1f877be8810c68] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058865 ~ 1729058889 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058880, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058880, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058880=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:08:19.537 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:19.540 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:19.541 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:08:19.579 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:08:19.579 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:08:19.659 INFO 4486 --- [ocp-async-10,7c8511078fc7ef2f,51125f446209ade7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:19.662 INFO 4486 --- [ocp-schedules-27,c50f5844651432c2,449099c92605c68a] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:08:19.663 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:19.994 INFO 4486 --- [prometheus-minute-schedule-1,a605ba7c53677c03,7153afc4a24d1223] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:20.000 INFO 4486 --- [prometheus-minute-schedule-1,a605ba7c53677c03,7153afc4a24d1223] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:20.161 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2024-10-16 14:08:20.161 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.ocp.core.property.PropertyService : Refresh config properties 2024-10-16 14:08:20.168 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 14:08:20.168 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 14:08:20.168 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 14:08:20.169 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 14:08:20.174 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : new: [] 2024-10-16 14:08:20.174 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : init spring 2024-10-16 14:08:20.174 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2024-10-16 14:08:20.174 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : starting 2024-10-16 14:08:20.220 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 14:08:20.228 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 14:08:20.228 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 14:08:20.228 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 14:08:20.228 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 14:08:20.228 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 14:08:20.235 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 14:08:20.663 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,39188b4dafc714a7,e0361d40909b0b9a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=39188b4dafc714a7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:20.663 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,39188b4dafc714a7,e0361d40909b0b9a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=39188b4dafc714a7, duration=2 ms] 2024-10-16 14:08:20.721 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2024-10-16 14:08:20.814 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 14:08:20.996 INFO 4486 --- [prometheus-cache-monitor1,8a252d3c691b9a69,704ac69ec7af5f1f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13856,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8187} 2024-10-16 14:08:20.996 INFO 4486 --- [prometheus-cache-monitor1,8a252d3c691b9a69,704ac69ec7af5f1f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:08:21.143 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.969563189S 2024-10-16 14:08:21.144 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2024-10-16 14:08:21.209 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2024-10-16 14:08:21.209 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2024-10-16 14:08:21.236 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2024-10-16 14:08:21.316 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2024-10-16 14:08:21.316 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2024-10-16 14:08:21.316 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2024-10-16 14:08:21.317 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.2 2024-10-16 14:08:21.317 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.2 2024-10-16 14:08:21.317 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2024-10-16 14:08:21.325 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2024-10-16 14:08:21.462 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] o.s.boot.SpringApplication : Started application in 1.293 seconds (JVM running for 79804.513) 2024-10-16 14:08:21.462 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.294049997S 2024-10-16 14:08:21.532 INFO 4486 --- [ocp-schedules-1,6b2ff590d4d3ed80,1556b076a1b83b8d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:08:21.659 INFO 4486 --- [ocp-async-1,8475ff1897f97c4a,86ca0c6d2cfeae53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:21.796 INFO 4486 --- [ocp-schedules-6,a1592439e4e18015,9af84593af162ad6] c.o.ocp.core.property.PropertyService : Refresh config properties done 2024-10-16 14:08:22.215 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0e63592c5cc60dbe,10669a9b5a4ab76e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=0e63592c5cc60dbe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:22.215 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,0e63592c5cc60dbe,10669a9b5a4ab76e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=0e63592c5cc60dbe, duration=2 ms] 2024-10-16 14:08:22.496 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e70b8dac3846b597,aff4457494534451] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e70b8dac3846b597, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:22.497 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e70b8dac3846b597,aff4457494534451] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e70b8dac3846b597, duration=1 ms] 2024-10-16 14:08:22.663 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:22.994 INFO 4486 --- [prometheus-minute-schedule-1,6121f8e1b786f3c4,8c29ac945aced3ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:23.000 INFO 4486 --- [prometheus-minute-schedule-1,6121f8e1b786f3c4,8c29ac945aced3ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:23.038 INFO 4486 --- [ocp-async-6,288798a4291450e9,1229050861004131] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:08:23.601 INFO 4486 --- [ocp-schedules-7,be97fb0199a0869a,a4af9d4f368a65b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:08:23.659 INFO 4486 --- [ocp-async-7,e2ca3bb0378542a6,10f70b03e3fa24d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:24.224 INFO 4486 --- [ocp-schedules-27,2d94a48e403e25cd,d6c21915449087da] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:08:25.449 INFO 4486 --- [ocp-schedules-5,0ffaa1e03d3f47a0,bb46f158eb0adf28] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:08:25.659 INFO 4486 --- [ocp-async-8,e82d07a538695432,eb0923fb5fdd2bc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:25.664 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:25.994 INFO 4486 --- [prometheus-minute-schedule-1,015a4a7805b51516,4ef3c7feb9357644] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:25.999 INFO 4486 --- [prometheus-minute-schedule-1,015a4a7805b51516,4ef3c7feb9357644] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:27.284 INFO 4486 --- [ocp-schedules-20,bcde0ca4d562c391,4b751d0188e3c2ac] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:08:27.284 INFO 4486 --- [ocp-schedules-20,bcde0ca4d562c391,4b751d0188e3c2ac] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:08:27.318 INFO 4486 --- [Collect all tenants major compactions2,bcde0ca4d562c391,3b0bb01f03350f5a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:08:27.318 INFO 4486 --- [Collect all tenants major compactions1,bcde0ca4d562c391,d4b7440d2b22efb2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:08:27.318 INFO 4486 --- [Collect all tenants major compactions5,bcde0ca4d562c391,b4e010f2288aa6e3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:08:27.318 INFO 4486 --- [Collect all tenants major compactions3,bcde0ca4d562c391,c689a3f1d64be501] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:08:27.319 INFO 4486 --- [Collect all tenants major compactions4,bcde0ca4d562c391,f6d1c5ed410b9eb0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:08:27.334 INFO 4486 --- [Collect all tenants major compactions2,bcde0ca4d562c391,3b0bb01f03350f5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:27.335 INFO 4486 --- [Collect all tenants major compactions2,bcde0ca4d562c391,3b0bb01f03350f5a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:08:27.337 INFO 4486 --- [Collect all tenants major compactions1,bcde0ca4d562c391,d4b7440d2b22efb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:27.338 INFO 4486 --- [Collect all tenants major compactions3,bcde0ca4d562c391,c689a3f1d64be501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:27.338 INFO 4486 --- [Collect all tenants major compactions1,bcde0ca4d562c391,d4b7440d2b22efb2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:08:27.340 INFO 4486 --- [Collect all tenants major compactions3,bcde0ca4d562c391,c689a3f1d64be501] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:08:27.341 INFO 4486 --- [Collect all tenants major compactions2,bcde0ca4d562c391,3b0bb01f03350f5a] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:08:27.342 INFO 4486 --- [Collect all tenants major compactions1,bcde0ca4d562c391,d4b7440d2b22efb2] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:08:27.343 INFO 4486 --- [Collect all tenants major compactions3,bcde0ca4d562c391,c689a3f1d64be501] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:08:27.348 INFO 4486 --- [Collect all tenants major compactions1,bcde0ca4d562c391,d4b7440d2b22efb2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:08:27.351 INFO 4486 --- [Collect all tenants major compactions2,bcde0ca4d562c391,3b0bb01f03350f5a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:08:27.351 INFO 4486 --- [Collect all tenants major compactions4,bcde0ca4d562c391,f6d1c5ed410b9eb0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:27.352 INFO 4486 --- [Collect all tenants major compactions5,bcde0ca4d562c391,b4e010f2288aa6e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:27.352 INFO 4486 --- [Collect all tenants major compactions3,bcde0ca4d562c391,c689a3f1d64be501] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:08:27.354 INFO 4486 --- [Collect all tenants major compactions4,bcde0ca4d562c391,f6d1c5ed410b9eb0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:08:27.354 INFO 4486 --- [Collect all tenants major compactions5,bcde0ca4d562c391,b4e010f2288aa6e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:08:27.357 INFO 4486 --- [Collect all tenants major compactions4,bcde0ca4d562c391,f6d1c5ed410b9eb0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:08:27.357 INFO 4486 --- [Collect all tenants major compactions4,bcde0ca4d562c391,f6d1c5ed410b9eb0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:08:27.359 INFO 4486 --- [Collect all tenants major compactions5,bcde0ca4d562c391,b4e010f2288aa6e3] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:08:27.359 INFO 4486 --- [Collect all tenants major compactions5,bcde0ca4d562c391,b4e010f2288aa6e3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:08:27.360 INFO 4486 --- [ocp-schedules-20,bcde0ca4d562c391,4b751d0188e3c2ac] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:08:27.660 INFO 4486 --- [ocp-async-8,0ab8d375a754ba9b,968512e1deee28cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:28.113 INFO 4486 --- [ocp-schedules-5,7c0abb7515e5d7ea,8e141c1efe2186a6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:08:28.113+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:08:28.115 INFO 4486 --- [ocp-schedules-5,7c0abb7515e5d7ea,8e141c1efe2186a6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:08:28.117 INFO 4486 --- [ocp-schedules-5,7c0abb7515e5d7ea,8e141c1efe2186a6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:08:28.118 INFO 4486 --- [ocp-schedules-5,7c0abb7515e5d7ea,8e141c1efe2186a6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:08:28.155 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,97d64e4280910e71,cf7aa5b09fbb4e27] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=97d64e4280910e71, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:08:28.155 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,97d64e4280910e71,cf7aa5b09fbb4e27] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=97d64e4280910e71, duration=1 ms] 2024-10-16 14:08:28.651 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a843514ca0a8fb0e,a36a5c49671ab2c8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a843514ca0a8fb0e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:28.651 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,a843514ca0a8fb0e,a36a5c49671ab2c8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a843514ca0a8fb0e, duration=2 ms] 2024-10-16 14:08:28.665 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:28.994 INFO 4486 --- [prometheus-minute-schedule-1,271302281868dbb6,4f374e59ccf07ec9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:29.002 INFO 4486 --- [prometheus-minute-schedule-1,271302281868dbb6,4f374e59ccf07ec9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:29.575 INFO 4486 --- [ocp-schedules-15,1c295c916484a386,053c01625c341154] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:08:29.644 WARN 4486 --- [scanner-13337630-1,1c295c916484a386,3e16986cd518b745] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:08:29.665 INFO 4486 --- [ocp-async-9,d12ce5256fc82989,d028139004ca2758] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:29.713 INFO 4486 --- [alarm-detect-1,1c295c916484a386,0a35abe45904dffe] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058880 ~ 1729058899 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058895, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058895, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058895=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:08:30.103 INFO 4486 --- [prometheus-second-schedule-1,03f281371ff00138,823001dce155df03] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:08:30.245 INFO 4486 --- [ocp-schedules-19,bd9aea81cda80200,3744961d100ce854] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 14:08:30.246 INFO 4486 --- [ocp-schedules-19,bd9aea81cda80200,3744961d100ce854] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:30.296 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:08:30.297 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.301 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:08:30.304 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:08:30.305 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.310 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:08:30.315 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.316 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:08:30.327 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.327 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:08:30.332 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.333 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:08:30.339 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.340 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:08:30.346 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.348 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:08:30.353 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.353 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:08:30.363 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:08:30.364 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.364 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.365 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:30.367 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.368 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:30.369 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:08:30.377 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.377 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:30.385 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.387 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:08:30.389 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.390 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:08:30.390 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.392 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:30.458 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.458 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:08:30.462 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.462 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.463 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:08:30.464 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:08:30.475 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.477 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:08:30.481 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.482 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:08:30.482 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.483 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.484 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.485 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.485 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.486 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.486 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.486 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:08:30.487 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.487 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.488 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.489 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.489 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.491 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.491 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.492 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.493 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.494 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.495 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.495 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.496 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.496 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.497 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.497 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:08:30.498 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.498 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.499 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.501 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.502 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 14:08:30.505 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.507 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.509 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.509 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:08:30.511 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.512 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.514 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:08:30.515 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.515 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.515 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:30.516 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.518 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.519 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.522 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.523 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.525 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.525 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.526 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:30.526 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.529 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.530 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.532 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:30.532 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.533 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:08:30.533 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.534 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.535 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.536 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:30.537 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.539 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.541 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 14:08:30.543 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.544 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 14:08:30.551 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 14:08:30.551 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:08:30.569 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.570 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.573 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.573 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:30.574 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.576 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.578 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.580 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.581 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.581 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.583 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.584 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.585 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:08:30.587 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.588 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.590 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.591 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:08:30.595 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.596 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:08:30.598 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.599 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.599 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 14:08:30.599 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.599 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 14:08:30.599 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.599 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 14:08:30.599 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:08:30.601 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 14:08:30.604 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 14:08:30.604 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 14:08:30.604 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:08:30.663 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:30.664 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:30.665 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:08:30.668 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,e45252d21f61b767,c6be23f5c0a67ce6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e45252d21f61b767, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:30.669 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,e45252d21f61b767,c6be23f5c0a67ce6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e45252d21f61b767, duration=2 ms] 2024-10-16 14:08:30.677 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.677 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 14:08:30.677 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.677 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 14:08:30.677 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:08:30.677 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 14:08:30.681 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 14:08:30.681 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:08:31.538 INFO 4486 --- [ocp-schedules-30,fbf09087fa3789d7,d6da21996388c98a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:08:31.662 INFO 4486 --- [ocp-async-7,c409d176f8ca0012,b5866f3887365747] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:31.666 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:31.994 INFO 4486 --- [prometheus-minute-schedule-1,1224fe76065ee704,1753615913d3442f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:32.000 INFO 4486 --- [prometheus-minute-schedule-1,1224fe76065ee704,1753615913d3442f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:32.220 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,cce471347cae86f3,96803b2299a2c81e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cce471347cae86f3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:32.220 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,cce471347cae86f3,96803b2299a2c81e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cce471347cae86f3, duration=2 ms] 2024-10-16 14:08:32.501 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,78f6324201125638,2b87e4894f5b5496] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=78f6324201125638, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:32.501 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,78f6324201125638,2b87e4894f5b5496] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=78f6324201125638, duration=1 ms] 2024-10-16 14:08:33.038 INFO 4486 --- [ocp-async-6,16686174bf81d158,7aacb8fa0ca7ec44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:08:33.662 INFO 4486 --- [ocp-async-8,5f99124e30e91e72,bdcfbafaa873bb53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:34.667 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:34.994 INFO 4486 --- [prometheus-minute-schedule-1,8a4a4bf9198fd2ce,4e98bc2a2c2098dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:35.002 INFO 4486 --- [prometheus-minute-schedule-1,8a4a4bf9198fd2ce,4e98bc2a2c2098dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:35.662 INFO 4486 --- [ocp-async-1,c40fc240d1edff6d,f40ab184281a8942] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:37.063 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:08:37.064 INFO 4486 --- [ocp-schedules-23,73acdb5bbf20a0bd,3143956b5a2601ae] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:08:37.066 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:08:37.067 INFO 4486 --- [ocp-schedules-14,5a8347d963d67061,c994d509cefeaa37] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:08:37.068 INFO 4486 --- [ocp-schedules-14,5a8347d963d67061,c994d509cefeaa37] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:37.070 INFO 4486 --- [ocp-schedules-23,73acdb5bbf20a0bd,3143956b5a2601ae] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:08:37.070 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:08:37.081 INFO 4486 --- [ocp-schedules-21,9d23ac41815a6292,99979e5120842553] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:08:37.083 INFO 4486 --- [ocp-schedules-21,9d23ac41815a6292,99979e5120842553] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:08:37.091 INFO 4486 --- [ocp-schedules-1,e3d4ccceb63f86df,aa34b35a67b9d9ec] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:08:37.091 INFO 4486 --- [ocp-schedules-30,dce6eb172c5e39df,904cd9ffba954e4a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:08:37.091 INFO 4486 --- [ocp-schedules-8,067ae7af1da31825,79289985b64644ec] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:08:37.092 INFO 4486 --- [host-check-agent-status1,dce6eb172c5e39df,d6e3bce7857e04a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:08:37.092 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:08:37.093 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:08:37.094 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:08:37.094 INFO 4486 --- [host-check-clockdiff1,e3d4ccceb63f86df,f032008faea75959] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:08:37.094 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:08:37.095 INFO 4486 --- [host-check-clockdiff2,e3d4ccceb63f86df,9cf19c0898672d3a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:08:37.096 INFO 4486 --- [host-check-agent-status2,dce6eb172c5e39df,a34f3d535d9e169e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:08:37.097 INFO 4486 --- [host-check-agent-status3,dce6eb172c5e39df,3ae8f2e6096bfbc9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:08:37.099 INFO 4486 --- [host-check-agent-status4,dce6eb172c5e39df,d2fbe6d660774d50] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:08:37.099 INFO 4486 --- [host-check-clockdiff3,e3d4ccceb63f86df,bde6bee41a56da38] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:08:37.100 INFO 4486 --- [host-check-clockdiff4,e3d4ccceb63f86df,db7b2b4bd59573c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:08:37.106 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:37.107 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:37.107 INFO 4486 --- [host-check-agent-status1,dce6eb172c5e39df,d6e3bce7857e04a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:37.107 INFO 4486 --- [host-check-agent-status2,dce6eb172c5e39df,a34f3d535d9e169e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:37.108 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:37.108 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:37.107 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:37.108 INFO 4486 --- [host-check-agent-status4,dce6eb172c5e39df,d2fbe6d660774d50] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:37.111 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:37.111 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:37.111 INFO 4486 --- [host-check-agent-status1,dce6eb172c5e39df,d6e3bce7857e04a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:37.141 INFO 4486 --- [host-check-agent-status4,dce6eb172c5e39df,d2fbe6d660774d50] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:37.141 INFO 4486 --- [host-check-agent-status3,dce6eb172c5e39df,3ae8f2e6096bfbc9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:37.141 INFO 4486 --- [host-check-agent-status2,dce6eb172c5e39df,a34f3d535d9e169e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:37.141 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:37.144 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:37.144 INFO 4486 --- [host-check-clockdiff3,e3d4ccceb63f86df,bde6bee41a56da38] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058917 4 4 , error= 2024-10-16 14:08:37.144 INFO 4486 --- [host-check-clockdiff1,e3d4ccceb63f86df,f032008faea75959] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058917 0 0 , error= 2024-10-16 14:08:37.150 INFO 4486 --- [host-check-agent-status3,dce6eb172c5e39df,3ae8f2e6096bfbc9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:37.148 INFO 4486 --- [host-check-clockdiff4,e3d4ccceb63f86df,db7b2b4bd59573c7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058917 4 4 , error= 2024-10-16 14:08:37.152 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.153 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:37.154 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:08:37.173 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:37.175 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.177 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.178 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:08:37.178 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:37.179 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:08:37.179 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:08:37.180 INFO 4486 --- [host-check-agent-status3,dce6eb172c5e39df,3ae8f2e6096bfbc9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:08:37.181 INFO 4486 --- [host-check-agent-status4,dce6eb172c5e39df,d2fbe6d660774d50] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:08:37.182 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:08:37.183 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:08:37.181953416+08:00 2024-10-16 14:08:37.183 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:08:37.186118730+08:00 2024-10-16 14:08:37.183 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:08:37.183 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:08:37.183+08:00 2024-10-16 14:08:37.183 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:08:37.183+08:00 2024-10-16 14:08:37.185 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:08:37.185 INFO 4486 --- [host-check-agent-status3,dce6eb172c5e39df,3ae8f2e6096bfbc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:08:37.185 INFO 4486 --- [host-check-agent-status3,dce6eb172c5e39df,3ae8f2e6096bfbc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:08:37.185 INFO 4486 --- [host-check-agent-status3,dce6eb172c5e39df,3ae8f2e6096bfbc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:08:37.186 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:08:37.189007184+08:00 2024-10-16 14:08:37.186 INFO 4486 --- [host-check-agent-status2,dce6eb172c5e39df,a34f3d535d9e169e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:08:37.187 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:08:37.186+08:00 2024-10-16 14:08:37.187 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:08:37.190910222+08:00 2024-10-16 14:08:37.187 INFO 4486 --- [host-check-agent-status4,dce6eb172c5e39df,d2fbe6d660774d50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:08:37.187 INFO 4486 --- [host-check-agent-status4,dce6eb172c5e39df,d2fbe6d660774d50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:08:37.187 INFO 4486 --- [host-check-agent-status4,dce6eb172c5e39df,d2fbe6d660774d50] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:08:37.188 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:08:37.187+08:00 2024-10-16 14:08:37.188 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.189 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:37.190 INFO 4486 --- [host-check-agent-status2,dce6eb172c5e39df,a34f3d535d9e169e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:08:37.190 INFO 4486 --- [host-check-agent-status2,dce6eb172c5e39df,a34f3d535d9e169e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:08:37.190 INFO 4486 --- [host-check-agent-status2,dce6eb172c5e39df,a34f3d535d9e169e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:08:37.192 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:08:37.183+08:00 and status AVAILABLE 2024-10-16 14:08:37.193 INFO 4486 --- [host-check-agent-status1,dce6eb172c5e39df,d6e3bce7857e04a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:08:37.194 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:08:37.195 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:08:37.187+08:00 and status AVAILABLE 2024-10-16 14:08:37.197 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:08:37.183+08:00 and status AVAILABLE 2024-10-16 14:08:37.198 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:08:37.199 INFO 4486 --- [host-check-agent-status1,dce6eb172c5e39df,d6e3bce7857e04a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:08:37.199 INFO 4486 --- [host-check-agent-status1,dce6eb172c5e39df,d6e3bce7857e04a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:08:37.199 INFO 4486 --- [host-check-agent-status1,dce6eb172c5e39df,d6e3bce7857e04a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:08:37.199 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.199 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.200 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:08:37.200 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:37.201 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:08:37.202 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.203 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:08:37.186+08:00 and status AVAILABLE 2024-10-16 14:08:37.203 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.204 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:08:37.205 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:37.207 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:37.207 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:37.209 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:37.209 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:08:37.209 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.210 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:08:37.212 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:37.212 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.213 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:08:37.213 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:37.215 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.215 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.215 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.215 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.215 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.215 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.218 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:37.219 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:37.222 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.223 INFO 4486 --- [ocp-schedules-10,aa29f6e4f70b63fa,7d4f76013738dfc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:08:37.233 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:37.237 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:37.240 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:37.243 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:08:37.246 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:08:37.261 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:08:37.273 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:08:37.276 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:08:37.276 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:08:37.287 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:08:37.287 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:08:37.303 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:37.308 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:37.309 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:37.319 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:08:37.330 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:08:37.335 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:08:37.342 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:08:37.347 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:08:37.349 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:08:37.353 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:08:37.358 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:08:37.362 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:08:37.370 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:37.382 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:37.384 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:37.400 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:08:37.413 INFO 4486 --- [host-check-host-status3,067ae7af1da31825,4c5a520d357ec11b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:08:37.415 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:08:37.419 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:08:37.425 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:08:37.429 INFO 4486 --- [host-check-host-status4,067ae7af1da31825,a10bee8a8b946602] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:08:37.433 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:08:37.440 INFO 4486 --- [host-check-host-status2,067ae7af1da31825,c068d01d673916e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:08:37.455 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:08:37.519 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:08:37.530 INFO 4486 --- [host-check-host-status1,067ae7af1da31825,b933a2e0a67891fd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:08:37.563 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:37.566 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.567 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:08:37.568 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.569 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.570 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:08:37.578 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.578 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.578 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.578 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.578 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.579 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.610 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:08:37.648 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:08:37.662 INFO 4486 --- [ocp-async-2,367dd4b4e6912ae2,623fa992a17b38a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:37.668 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:37.702 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:37.721 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:37.722 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.723 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:08:37.724 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:08:37.730 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.731 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:08:37.732 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.732 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.733 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:08:37.744 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.744 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.744 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.744 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.744 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:37.745 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:37.752 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:08:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:08:37.977 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:08:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:08:37.994 INFO 4486 --- [prometheus-minute-schedule-1,36c4fdb405534f17,70b97a84c8e4bb2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:38.001 INFO 4486 --- [prometheus-minute-schedule-1,36c4fdb405534f17,70b97a84c8e4bb2f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:38.035 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:38.037 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:38.038 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:08:38.040 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.040 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:38.041 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:08:38.052 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.052 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.052 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.052 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.052 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.053 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:38.059 INFO 4486 --- [host-check-clockdiff2,e3d4ccceb63f86df,9cf19c0898672d3a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058918 4 4 , error= 2024-10-16 14:08:38.076 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:08:38.301 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:08:38.351 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:38.353 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:38.354 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:08:38.358 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.359 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:38.360 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:08:38.371 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.371 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.371 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.371 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.371 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:08:38.373 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:38.442 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:08:38.471 INFO 4486 --- [ocp-schedules-13,8fdec1507c476f24,2c97c715154756a1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [AlarmTargetStat(alarmType=base_backup_too_long_time_no_one_success_task, target=alarm_template_id=0:ob_cluster=obce_3zones-1701313639:tenant_name=yewu02, statCount=1)] 2024-10-16 14:08:38.471 INFO 4486 --- [ocp-schedules-13,8fdec1507c476f24,2c97c715154756a1] c.o.o.a.service.OcpAlarmSearchService : obce_3zones-1701313639 2024-10-16 14:08:38.475 INFO 4486 --- [ocp-schedules-13,8fdec1507c476f24,2c97c715154756a1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2024-10-16 14:08:38.657 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3c0558359837b111,69ad7759c172a496] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3c0558359837b111, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:38.657 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,3c0558359837b111,69ad7759c172a496] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3c0558359837b111, duration=2 ms] 2024-10-16 14:08:38.671 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:08:38.446265, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:08:38.672 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:08:38.446265, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:08:38.687 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,17cb02953503974f,81b0abef44df05b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=17cb02953503974f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:08:38.688 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,17cb02953503974f,81b0abef44df05b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=17cb02953503974f, duration=2 ms] 2024-10-16 14:08:38.720 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:08:38.446265} } 2024-10-16 14:08:38.806 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:08:38.820 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:08:38.823 INFO 4486 --- [ocp-schedules-9,f15e6996fcc0d7e4,2c106d436ef2ba33] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:08:39.663 INFO 4486 --- [ocp-async-6,c934e2fd19efc56c,e7cdfb92a8c05d18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:39.666 INFO 4486 --- [ocp-schedules-24,49877d3bc5ab1b30,9f0d9dc5bd86fde8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:08:39.813 INFO 4486 --- [ocp-schedules-17,126dae81d6db641b,a675f4b8a737fe10] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:08:39.858 WARN 4486 --- [scanner-13337946-1,126dae81d6db641b,5f9a3ffe59e1d342] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:08:40.006 INFO 4486 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:08:40.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:08:40.026 INFO 4486 --- [Check obproxy2,39f03add7a3672e9,2d2e819a0eff2c83] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:08:40.028 INFO 4486 --- [Check obproxy1,b6a4aeb495b6928c,ad51c6c54f8cbcbc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:08:40.031 INFO 4486 --- [Collect obproxy2,95f02a845be94f52,31fd2edf846f880b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:08:40.032 INFO 4486 --- [Collect obproxy1,b00a2e8a69d87047,e9483ca3bf5a3399] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:08:40.038 INFO 4486 --- [Check obproxy2,39f03add7a3672e9,2d2e819a0eff2c83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:40.042 INFO 4486 --- [Check obproxy1,b6a4aeb495b6928c,ad51c6c54f8cbcbc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:40.043 INFO 4486 --- [Collect obproxy2,95f02a845be94f52,31fd2edf846f880b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:40.044 INFO 4486 --- [Collect obproxy1,b00a2e8a69d87047,e9483ca3bf5a3399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:08:40.073 INFO 4486 --- [Check obproxy2,39f03add7a3672e9,2d2e819a0eff2c83] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:08:40.073 INFO 4486 --- [Check obproxy2,39f03add7a3672e9,2d2e819a0eff2c83] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:08:40.078 INFO 4486 --- [Collect obproxy2,95f02a845be94f52,31fd2edf846f880b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:08:40.078 INFO 4486 --- [Collect obproxy1,b00a2e8a69d87047,e9483ca3bf5a3399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:08:40.081 INFO 4486 --- [Collect obproxy1,b00a2e8a69d87047,e9483ca3bf5a3399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:08:40.084 INFO 4486 --- [Collect obproxy2,95f02a845be94f52,31fd2edf846f880b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:08:40.085 INFO 4486 --- [Collect obproxy1,b00a2e8a69d87047,e9483ca3bf5a3399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:08:40.086 INFO 4486 --- [Check obproxy1,b6a4aeb495b6928c,ad51c6c54f8cbcbc] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:08:40.086 INFO 4486 --- [Check obproxy1,b6a4aeb495b6928c,ad51c6c54f8cbcbc] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:08:40.088 INFO 4486 --- [Collect obproxy2,95f02a845be94f52,31fd2edf846f880b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:08:40.669 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:40.672 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,414afbdaf7cf1f4f,b7a95c1470218c27] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=414afbdaf7cf1f4f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:40.673 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,414afbdaf7cf1f4f,b7a95c1470218c27] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=414afbdaf7cf1f4f, duration=1 ms] 2024-10-16 14:08:40.994 INFO 4486 --- [prometheus-minute-schedule-1,f1d5d28573e8ae49,39d4bfe3e534002f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:40.995 INFO 4486 --- [prometheus-hour-schedule-1,881c9308324c2097,ae4029bb031a597b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:08:40.999 INFO 4486 --- [prometheus-minute-schedule-1,f1d5d28573e8ae49,39d4bfe3e534002f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:41.543 INFO 4486 --- [ocp-schedules-20,ed4d80bf3d61eeef,a07c45d07f4002f4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:08:41.663 INFO 4486 --- [ocp-async-5,49f7a7313b8e4122,5c34a3b2207f99a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:42.208 INFO 4486 --- [ocp-schedules-6,8d3f2b63d625cac9,1010fcaf64931c78] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:08:42.224 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,736921f66bd587ef,126c1949f3d9adc6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=736921f66bd587ef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:42.225 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,736921f66bd587ef,126c1949f3d9adc6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=736921f66bd587ef, duration=1 ms] 2024-10-16 14:08:42.505 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,a05ed3d43316208a,2e87aec87c6ffa79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a05ed3d43316208a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:42.506 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,a05ed3d43316208a,2e87aec87c6ffa79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a05ed3d43316208a, duration=2 ms] 2024-10-16 14:08:43.038 INFO 4486 --- [ocp-async-8,e327119f09e1f320,b733d528aa76bffb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:08:43.663 INFO 4486 --- [ocp-async-2,569648f10c68fbc5,f316372a8ac28ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:43.670 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:43.994 INFO 4486 --- [prometheus-minute-schedule-1,232048bd91f84138,6753bf3081556d50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:44.000 INFO 4486 --- [prometheus-minute-schedule-1,232048bd91f84138,6753bf3081556d50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:45.641 INFO 4486 --- [ocp-schedules-27,3546de6b032fda7c,9c6fbbee8665c3cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:08:45.657 INFO 4486 --- [ocp-schedules-27,3546de6b032fda7c,9c6fbbee8665c3cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:08:45.657 INFO 4486 --- [ocp-schedules-27,3546de6b032fda7c,9c6fbbee8665c3cb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:08:45.663 INFO 4486 --- [ocp-async-8,34ec6d7e93632f6a,fe60721d0f9b52db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:45.670 INFO 4486 --- [ocp-schedules-5,7c294b3a5bdced16,90b1ba4af884e6a6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:08:45.670 INFO 4486 --- [ocp-schedules-5,7c294b3a5bdced16,90b1ba4af884e6a6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:08:46.493 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,bc8f14a249343628,68b5b15f4ee65ddf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=bc8f14a249343628, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:08:46.493 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,bc8f14a249343628,68b5b15f4ee65ddf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=bc8f14a249343628, duration=2 ms] 2024-10-16 14:08:46.671 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:46.994 INFO 4486 --- [prometheus-minute-schedule-1,244c6b138384ee04,b609925517aad0a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:47.001 INFO 4486 --- [prometheus-minute-schedule-1,244c6b138384ee04,b609925517aad0a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:47.070 INFO 4486 --- [ocp-schedules-14,5a8347d963d67061,c994d509cefeaa37] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:08:47.183 INFO 4486 --- [ocp-schedules-21,9d23ac41815a6292,99979e5120842553] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:08:47.327 INFO 4486 --- [ocp-schedules-24,cbda671fabb459dc,85bfe6f745f4e5cb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:08:47.663 INFO 4486 --- [ocp-async-9,7695936b531a5bca,9c8ede8c641fb36b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:48.664 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,56ce66a6b57743c6,458bd28e3cd08291] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=56ce66a6b57743c6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:48.664 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,56ce66a6b57743c6,458bd28e3cd08291] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=56ce66a6b57743c6, duration=2 ms] 2024-10-16 14:08:49.579 INFO 4486 --- [ocp-schedules-16,9ae032ef7ef3f0b3,2cc5ccd40a439c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:08:49.580 INFO 4486 --- [ocp-schedules-16,e465af81f31f389e,95655a9c3e3b598c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:08:49.581 INFO 4486 --- [ocp-schedules-16,e465af81f31f389e,95655a9c3e3b598c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:08:49.621 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.622 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:49.624 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.625 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:49.650 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.651 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:49.664 INFO 4486 --- [ocp-async-7,fb5f463910ddbb18,9f57601759def2ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:49.665 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:49.671 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:49.671 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.672 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:08:49.704 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:49.705 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.706 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:49.707 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:08:49.713 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.716 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:08:49.717 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:49.732 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:49.732 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:49.748 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:49.757 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:49.758 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.758 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:49.759 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:08:49.768 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:08:49.768 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:49.770 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.770 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:08:49.774 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:49.777 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:49.804 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:08:49.819 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:08:49.819 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058340307000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:08:49.829 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:49.831 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.832 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:08:49.836 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.839 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:08:49.843 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:49.869 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:49.869 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:49.896 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:08:49.902 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:49.902 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:49.906 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:49.910 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:49.911 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:08:49.922 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:08:49.922 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058886919000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:08:49.927 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:49.927 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:49.954 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:49.954 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:08:49.963 INFO 4486 --- [ocp-schedules-20,0635db57b1dce480,79163c18f86b4ce9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:08:49.983 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:49.983 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:08:49.995 INFO 4486 --- [prometheus-minute-schedule-1,0326ffca658be22f,5126333c9ce5da90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:50.002 WARN 4486 --- [scanner-13338182-1,0635db57b1dce480,b7be45f80a9d09c3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:08:50.005 INFO 4486 --- [prometheus-minute-schedule-1,0326ffca658be22f,5126333c9ce5da90] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:50.016 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:08:50.048 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:50.050 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:50.051 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:08:50.057 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:08:50.063 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:08:50.069 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:08:50.070 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:08:50.078 INFO 4486 --- [alarm-detect-9,0635db57b1dce480,9cffccd9383fa5ea] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058895 ~ 1729058920 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058910, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058910, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058910=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:08:50.099 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:08:50.237 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:08:50.265 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:08:50.266 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:08:50.267 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:08:50.298 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:08:50.298 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:08:50.611 INFO 4486 --- [ocp-schedules-3,c5968d2d4030a572,5a7c533514cefd1a] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:08:50.611 INFO 4486 --- [ocp-schedules-3,c5968d2d4030a572,5a7c533514cefd1a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:08:50.616 INFO 4486 --- [ocp-schedules-3,c5968d2d4030a572,5a7c533514cefd1a] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:08:50.616 INFO 4486 --- [ocp-schedules-3,c5968d2d4030a572,5a7c533514cefd1a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:08:50.621 INFO 4486 --- [ocp-schedules-3,c5968d2d4030a572,5a7c533514cefd1a] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:08:50.621 INFO 4486 --- [ocp-schedules-3,c5968d2d4030a572,5a7c533514cefd1a] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703046246634950656 2024-10-16 14:08:50.677 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5b6c93f402ab29ce,9c0e88119ee68bbb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5b6c93f402ab29ce, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:50.677 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,5b6c93f402ab29ce,9c0e88119ee68bbb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5b6c93f402ab29ce, duration=1 ms] 2024-10-16 14:08:51.548 INFO 4486 --- [ocp-schedules-27,8c967fd58c2cb1b7,e244d4a20bfea616] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:08:51.664 INFO 4486 --- [ocp-async-7,207aef4617ea6a66,699d0f7feecad54b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:51.879 INFO 4486 --- [ocp-schedules-30,46129d31704cac6c,5b7733515f5b3343] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:08:51.879 INFO 4486 --- [ocp-schedules-30,46129d31704cac6c,5b7733515f5b3343] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:08:51.879 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:08:51.879 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:08:51.879 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:08:51.879 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:08:51.879 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:08:51.884 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:08:51.884 INFO 4486 --- [ocp-schedules-30,46129d31704cac6c,5b7733515f5b3343] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:08:51.884 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:08:51.884 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:08:51.884 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:08:51.885 INFO 4486 --- [ocp-schedules-30,46129d31704cac6c,5b7733515f5b3343] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:08:51.886 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:08:51.886 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:08:51.886 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:08:51.886 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:08:51.886 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:08:51.887 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:08:51.909 INFO 4486 --- [ocp-schedules-30,46129d31704cac6c,5b7733515f5b3343] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:08:51.909 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:08:51.910 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:08:51.911 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:08:51.914 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:08:51.915 INFO 4486 --- [ForkJoinPool.commonPool-worker-13,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:08:51.920 INFO 4486 --- [ocp-schedules-30,46129d31704cac6c,5b7733515f5b3343] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:08:52.229 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,3f30de7cad02797a,7dfa36dd76427daf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=3f30de7cad02797a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:52.229 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,3f30de7cad02797a,7dfa36dd76427daf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=3f30de7cad02797a, duration=1 ms] 2024-10-16 14:08:52.510 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,573b64fc057722f9,d2cd3faae1a71745] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=573b64fc057722f9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:52.510 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,573b64fc057722f9,d2cd3faae1a71745] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=573b64fc057722f9, duration=1 ms] 2024-10-16 14:08:52.673 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:52.994 INFO 4486 --- [prometheus-minute-schedule-1,1be1b297335cf50c,a2e0140e270b2d23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:53.001 INFO 4486 --- [prometheus-minute-schedule-1,1be1b297335cf50c,a2e0140e270b2d23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:53.038 INFO 4486 --- [ocp-async-10,3ac9618fd60b26a1,cde3bbf66f0a63f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:08:53.664 INFO 4486 --- [ocp-async-5,ea3f4c332602fa8c,922e20b7de5d306b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:54.886 INFO 4486 --- [ocp-schedules-27,a3b636eb8572f333,facbd28a0725d876] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2024-10-16T14:08:54.886+08:00, tenantSize=5 2024-10-16 14:08:54.892 INFO 4486 --- [ocp-schedules-27,a3b636eb8572f333,facbd28a0725d876] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67currentTimeUs:1729058934887000 2024-10-16 14:08:54.892 INFO 4486 --- [ocp-schedules-27,a3b636eb8572f333,facbd28a0725d876] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67diagnosisStartTimeUs:1729058634806000 2024-10-16 14:08:54.909 INFO 4486 --- [ocp-schedules-27,a3b636eb8572f333,facbd28a0725d876] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_5_67totalLength:0 2024-10-16 14:08:54.915 INFO 4486 --- [ocp-schedules-27,a3b636eb8572f333,facbd28a0725d876] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015currentTimeUs:1729058934912000 2024-10-16 14:08:54.915 INFO 4486 --- [ocp-schedules-27,a3b636eb8572f333,facbd28a0725d876] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015diagnosisStartTimeUs:1729058634841000 2024-10-16 14:08:54.944 INFO 4486 --- [ocp-schedules-27,a3b636eb8572f333,facbd28a0725d876] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1000003_1000015totalLength:0 2024-10-16 14:08:55.664 INFO 4486 --- [ocp-async-2,b0c915a204f1e0bf,ff3dc89b95673394] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:55.673 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:55.994 INFO 4486 --- [prometheus-minute-schedule-1,4103fa77d693e359,d639c0abbeb62ddf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:56.000 INFO 4486 --- [prometheus-minute-schedule-1,4103fa77d693e359,d639c0abbeb62ddf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:57.665 INFO 4486 --- [ocp-async-8,4f14ae1d4dcd406f,e5539ba55e92ef5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:58.670 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,7f893c1adc108b50,57487508ff4dc285] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=7f893c1adc108b50, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:08:58.671 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,7f893c1adc108b50,57487508ff4dc285] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=7f893c1adc108b50, duration=1 ms] 2024-10-16 14:08:58.675 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:08:58.994 INFO 4486 --- [prometheus-minute-schedule-1,60a28726205a5ad0,ea6dfe6922a02d8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729058940, interval=60 2024-10-16 14:08:59.000 INFO 4486 --- [prometheus-minute-schedule-1,60a28726205a5ad0,ea6dfe6922a02d8e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:08:59.665 INFO 4486 --- [ocp-async-1,6f1fd3f3d1a32513,9bda3430e6e4f9c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:08:59.671 INFO 4486 --- [ocp-schedules-29,2bc008ad88a8fd64,3d5acb1319052333] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:09:00.156 INFO 4486 --- [ocp-schedules-18,b58807b5d8e2a3d8,c8b4fb7ccc600098] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:09:00.195 WARN 4486 --- [scanner-13338506-1,b58807b5d8e2a3d8,aa2f09de107845cb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:09:00.248 INFO 4486 --- [alarm-detect-4,b58807b5d8e2a3d8,b356c8daef376c0d] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058910 ~ 1729058930 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058925, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058925, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058925=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:09:00.310 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,92e2f8c2172ed268,35ee8d2df7859733] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=obce_3zones&version=2&ObClusterId=1701313639, client=10.197.49.176, traceId=92e2f8c2172ed268, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=obce_3zones,1701313639,2,{"ObRegion":"obce_3zones","ObCluster":"obce_3zones","ObRegionId":1701313639,"ObClusterId":1701313639,"Type":"PRIMARY","timestamp":1729058940307591,"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2024-10-16 14:09:00.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,92e2f8c2172ed268,35ee8d2df7859733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=obce_3zones, obClusterId=1701313639, rootServerJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, groupName=obce_3zones, obproxyClusters=null, tenantInfos=[]) into config server cache 2024-10-16 14:09:00.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,92e2f8c2172ed268,35ee8d2df7859733] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2024-10-16 14:09:00.313 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,92e2f8c2172ed268,35ee8d2df7859733] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=obce_3zones, clusterId=1701313639, affectRows=1 2024-10-16 14:09:00.314 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,92e2f8c2172ed268,35ee8d2df7859733] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.197.49.176, traceId=92e2f8c2172ed268, duration=7 ms] 2024-10-16 14:09:00.727 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,b8a382b76b5e2edb,fe06fabf36373e69] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b8a382b76b5e2edb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:00.727 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,b8a382b76b5e2edb,fe06fabf36373e69] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b8a382b76b5e2edb, duration=1 ms] 2024-10-16 14:09:01.552 INFO 4486 --- [ocp-schedules-25,0ac61bb90dc4d78a,b1c21b5e492fa814] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:09:01.665 INFO 4486 --- [ocp-async-9,b9649a559b49b712,b94721659edefdf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:01.675 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:01.994 INFO 4486 --- [prometheus-minute-schedule-1,e0afdb2f5b75e6c4,eb3f4f81f4ce0952] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:02.000 INFO 4486 --- [prometheus-minute-schedule-1,e0afdb2f5b75e6c4,eb3f4f81f4ce0952] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:02.234 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,413719500b6eb028,61c022dcdc3f3191] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=413719500b6eb028, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:02.234 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,413719500b6eb028,61c022dcdc3f3191] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=413719500b6eb028, duration=1 ms] 2024-10-16 14:09:02.515 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,32c09c1dbb2ad833,1e64c9a301e3ed8b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=32c09c1dbb2ad833, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:02.515 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,32c09c1dbb2ad833,1e64c9a301e3ed8b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=32c09c1dbb2ad833, duration=1 ms] 2024-10-16 14:09:03.039 INFO 4486 --- [ocp-async-4,ef4448cee5a8cf8e,4ebb7d02ac922472] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:09:03.127 INFO 4486 --- [ocp-schedules-11,377dc3afe1d566df,b9a1106a3125d835] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:09:03.127+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:09:03.131 INFO 4486 --- [ocp-schedules-11,377dc3afe1d566df,b9a1106a3125d835] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:09:03.141 INFO 4486 --- [ocp-schedules-11,377dc3afe1d566df,b9a1106a3125d835] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:09:03.152 INFO 4486 --- [ocp-schedules-11,377dc3afe1d566df,b9a1106a3125d835] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:09:03.665 INFO 4486 --- [ocp-async-3,cfc0e6927275f2a5,8769274208d2680b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:04.677 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:04.994 INFO 4486 --- [prometheus-minute-schedule-1,35a217a8e179a0fd,d02a83db66504dd0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:05.001 INFO 4486 --- [prometheus-minute-schedule-1,35a217a8e179a0fd,d02a83db66504dd0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:05.665 INFO 4486 --- [ocp-async-4,1aa39ebea17aff43,fb20fd37617e3387] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:07.066 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:09:07.066 INFO 4486 --- [ocp-schedules-18,23846fe18dadd3e9,5c2370993ffac276] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:09:07.069 INFO 4486 --- [ocp-schedules-18,23846fe18dadd3e9,5c2370993ffac276] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:09:07.081 INFO 4486 --- [ocp-schedules-7,09f33a62a6afdf07,4e0772a7569f9cd4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:09:07.083 INFO 4486 --- [ocp-schedules-7,09f33a62a6afdf07,4e0772a7569f9cd4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:09:07.085 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:09:07.091 INFO 4486 --- [ocp-schedules-23,640ba308834aa929,d22981924fcb0c52] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:09:07.091 INFO 4486 --- [ocp-schedules-9,7b906952aa0aad8a,ecb1b5aa16625044] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:09:07.092 INFO 4486 --- [ocp-schedules-21,e6c3a0449deb9523,a08767668c4ac146] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:09:07.092 INFO 4486 --- [host-check-agent-status1,640ba308834aa929,3c40b96593956948] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:09:07.092 INFO 4486 --- [host-check-clockdiff1,7b906952aa0aad8a,00eb314e60b75b11] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:09:07.093 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:09:07.093 INFO 4486 --- [host-check-clockdiff2,7b906952aa0aad8a,4a9640eb194faa5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:09:07.093 INFO 4486 --- [host-check-agent-status2,640ba308834aa929,ccb604596363b1a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:09:07.093 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:09:07.094 INFO 4486 --- [host-check-agent-status3,640ba308834aa929,67d6fd84e7c6dc34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:09:07.094 INFO 4486 --- [host-check-agent-status4,640ba308834aa929,e533b222e27680a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:09:07.095 INFO 4486 --- [host-check-clockdiff3,7b906952aa0aad8a,6f1eeb4313e62fbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:09:07.095 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:09:07.096 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:09:07.096 INFO 4486 --- [host-check-clockdiff4,7b906952aa0aad8a,51442aac7e3e7a0b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:09:07.102 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:09:07.102 INFO 4486 --- [host-check-agent-status1,640ba308834aa929,3c40b96593956948] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:07.103 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:07.103 INFO 4486 --- [host-check-agent-status3,640ba308834aa929,67d6fd84e7c6dc34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:09:07.103 INFO 4486 --- [host-check-agent-status2,640ba308834aa929,ccb604596363b1a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:09:07.103 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:09:07.103 INFO 4486 --- [host-check-agent-status4,640ba308834aa929,e533b222e27680a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:09:07.104 INFO 4486 --- [host-check-agent-status1,640ba308834aa929,3c40b96593956948] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:07.104 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:09:07.105 INFO 4486 --- [host-check-agent-status2,640ba308834aa929,ccb604596363b1a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:09:07.117 INFO 4486 --- [host-check-agent-status3,640ba308834aa929,67d6fd84e7c6dc34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:09:07.128 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:09:07.128 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:07.129 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:09:07.129 INFO 4486 --- [host-check-agent-status4,640ba308834aa929,e533b222e27680a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:09:07.143 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:09:07.141 INFO 4486 --- [host-check-clockdiff1,7b906952aa0aad8a,00eb314e60b75b11] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058947 0 0 , error= 2024-10-16 14:09:07.147 INFO 4486 --- [host-check-clockdiff2,7b906952aa0aad8a,4a9640eb194faa5c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058947 4 4 , error= 2024-10-16 14:09:07.154 INFO 4486 --- [host-check-clockdiff4,7b906952aa0aad8a,51442aac7e3e7a0b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058947 3 3 , error= 2024-10-16 14:09:07.155 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.156 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:07.157 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:09:07.178 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.179 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:07.179 INFO 4486 --- [host-check-agent-status4,640ba308834aa929,e533b222e27680a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:09:07.179 INFO 4486 --- [host-check-agent-status3,640ba308834aa929,67d6fd84e7c6dc34] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:09:07.180 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:09:07.181 INFO 4486 --- [host-check-agent-status2,640ba308834aa929,ccb604596363b1a7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:09:07.181 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:09:07.181 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:09:07.183 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:09:07.183 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:09:07.184 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:09:07.187249244+08:00 2024-10-16 14:09:07.184 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:09:07.183497477+08:00 2024-10-16 14:09:07.184 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:09:07.186389036+08:00 2024-10-16 14:09:07.184 INFO 4486 --- [host-check-agent-status1,640ba308834aa929,3c40b96593956948] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:09:07.184 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:09:07.184+08:00 2024-10-16 14:09:07.184 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:09:07.184+08:00 2024-10-16 14:09:07.184 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:09:07.184+08:00 2024-10-16 14:09:07.185 INFO 4486 --- [host-check-agent-status3,640ba308834aa929,67d6fd84e7c6dc34] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:09:07.185 INFO 4486 --- [host-check-agent-status3,640ba308834aa929,67d6fd84e7c6dc34] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:09:07.185 INFO 4486 --- [host-check-agent-status3,640ba308834aa929,67d6fd84e7c6dc34] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:09:07.186 INFO 4486 --- [host-check-agent-status2,640ba308834aa929,ccb604596363b1a7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:09:07.186 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:09:07.189929133+08:00 2024-10-16 14:09:07.186 INFO 4486 --- [host-check-agent-status2,640ba308834aa929,ccb604596363b1a7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:09:07.186 INFO 4486 --- [host-check-agent-status2,640ba308834aa929,ccb604596363b1a7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:09:07.186 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:09:07.186+08:00 2024-10-16 14:09:07.187 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:09:07.184+08:00 and status AVAILABLE 2024-10-16 14:09:07.187 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:09:07.184+08:00 and status AVAILABLE 2024-10-16 14:09:07.187 INFO 4486 --- [host-check-agent-status4,640ba308834aa929,e533b222e27680a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:09:07.187 INFO 4486 --- [host-check-agent-status4,640ba308834aa929,e533b222e27680a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:09:07.187 INFO 4486 --- [host-check-agent-status4,640ba308834aa929,e533b222e27680a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:09:07.188 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:09:07.184+08:00 and status AVAILABLE 2024-10-16 14:09:07.189 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:09:07.186+08:00 and status AVAILABLE 2024-10-16 14:09:07.189 INFO 4486 --- [host-check-agent-status1,640ba308834aa929,3c40b96593956948] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:09:07.189 INFO 4486 --- [host-check-agent-status1,640ba308834aa929,3c40b96593956948] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:09:07.189 INFO 4486 --- [host-check-agent-status1,640ba308834aa929,3c40b96593956948] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:09:07.189 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:09:07.189 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:09:07.190 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:09:07.190 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:09:07.196 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.198 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:07.198 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:09:07.198 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:09:07.199 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:09:07.199 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:07.200 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:09:07.200 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:09:07.200 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:09:07.203 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:09:07.212 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.213 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:09:07.223 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:07.223 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.224 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:07.224 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:09:07.224 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:07.232 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:07.234 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.235 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:09:07.238 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.239 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:09:07.247 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:07.248 INFO 4486 --- [ocp-schedules-17,9c0339cfbba7e494,4703f3a8af30643f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:09:07.261 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:09:07.261 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:09:07.261 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:09:07.271 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:09:07.272 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:09:07.274 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:09:07.274 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:09:07.288 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:09:07.289 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:07.289 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:07.291 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:07.308 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:07.321 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:09:07.331 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:09:07.333 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:09:07.334 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:09:07.341 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:09:07.346 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:09:07.349 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:07.357 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:07.360 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:07.366 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:09:07.377 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:09:07.378 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:09:07.386 INFO 4486 --- [host-check-host-status3,e6c3a0449deb9523,8740a5c2eb5d1449] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:09:07.388 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:09:07.388 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:09:07.393 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:07.400 INFO 4486 --- [host-check-host-status2,e6c3a0449deb9523,92ede241d50597bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:09:07.401 INFO 4486 --- [host-check-host-status4,e6c3a0449deb9523,c2500e11d96a5192] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:09:07.443 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:09:07.455 INFO 4486 --- [host-check-host-status1,e6c3a0449deb9523,ce509348c6e0ed27] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:09:07.666 INFO 4486 --- [ocp-async-10,8ae10cb3aba7eb83,7132b546f9810585] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:07.678 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:07.994 INFO 4486 --- [prometheus-minute-schedule-1,12683393664b0df0,5655319c0e8f409c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:08.000 INFO 4486 --- [prometheus-minute-schedule-1,12683393664b0df0,5655319c0e8f409c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:08.058 INFO 4486 --- [host-check-clockdiff3,7b906952aa0aad8a,6f1eeb4313e62fbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058948 4 4 , error= 2024-10-16 14:09:08.676 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,4ec8faea5f97b96f,d29633ec7789639d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=4ec8faea5f97b96f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:08.676 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,4ec8faea5f97b96f,d29633ec7789639d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=4ec8faea5f97b96f, duration=2 ms] 2024-10-16 14:09:09.666 INFO 4486 --- [ocp-async-2,3f6e8a7b1cd37af4,f2c32c7652be687b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:10.006 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:09:10.006 INFO 4486 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:09:10.025 INFO 4486 --- [Check obproxy2,b6ea71120c8066de,6c1d079ae184bc1f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:09:10.026 INFO 4486 --- [Check obproxy2,b6ea71120c8066de,6c1d079ae184bc1f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:10.026 INFO 4486 --- [Check obproxy1,051262aaeeb099bd,13391417fad046c6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:09:10.026 INFO 4486 --- [Check obproxy1,051262aaeeb099bd,13391417fad046c6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:10.029 INFO 4486 --- [Collect obproxy1,7d7554a9a10ca9e9,83926a9b1d9113e7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:09:10.029 INFO 4486 --- [Collect obproxy1,7d7554a9a10ca9e9,83926a9b1d9113e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:10.029 INFO 4486 --- [Collect obproxy2,5c352c302f8fec33,68a1f876f1f3eb41] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:09:10.029 INFO 4486 --- [Collect obproxy2,5c352c302f8fec33,68a1f876f1f3eb41] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:10.050 INFO 4486 --- [Check obproxy2,b6ea71120c8066de,6c1d079ae184bc1f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:09:10.050 INFO 4486 --- [Check obproxy2,b6ea71120c8066de,6c1d079ae184bc1f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:09:10.050 INFO 4486 --- [Collect obproxy2,5c352c302f8fec33,68a1f876f1f3eb41] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:09:10.054 INFO 4486 --- [Check obproxy1,051262aaeeb099bd,13391417fad046c6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:09:10.054 INFO 4486 --- [Check obproxy1,051262aaeeb099bd,13391417fad046c6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:09:10.055 INFO 4486 --- [Collect obproxy2,5c352c302f8fec33,68a1f876f1f3eb41] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:09:10.058 INFO 4486 --- [Collect obproxy1,7d7554a9a10ca9e9,83926a9b1d9113e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:09:10.060 INFO 4486 --- [Collect obproxy2,5c352c302f8fec33,68a1f876f1f3eb41] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:09:10.061 INFO 4486 --- [Collect obproxy1,7d7554a9a10ca9e9,83926a9b1d9113e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:09:10.064 INFO 4486 --- [Collect obproxy1,7d7554a9a10ca9e9,83926a9b1d9113e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:09:10.316 INFO 4486 --- [ocp-schedules-14,9d252a44be921720,7377f5c44da6cb60] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:09:10.340 WARN 4486 --- [scanner-13338724-1,9d252a44be921720,a3041f7f59c81bc0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:09:10.446 INFO 4486 --- [alarm-detect-12,9d252a44be921720,57d9fac7885e0bd5] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058925 ~ 1729058940 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058940, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058940, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058940=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:09:10.678 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:10.732 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d9c46ed73d276bbd,d0f3c265c24502f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d9c46ed73d276bbd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:10.733 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d9c46ed73d276bbd,d0f3c265c24502f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d9c46ed73d276bbd, duration=1 ms] 2024-10-16 14:09:10.981 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:09:10.994 INFO 4486 --- [prometheus-minute-schedule-1,2cdfe985ab8d562e,49c1c5c77db50706] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:11.000 INFO 4486 --- [prometheus-minute-schedule-1,2cdfe985ab8d562e,49c1c5c77db50706] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:11.004 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:09:11.558 INFO 4486 --- [ocp-schedules-1,cd49ad4ab7831165,6189507b118240ab] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:09:11.666 INFO 4486 --- [ocp-async-3,0f7e46d1d5486283,aad5d9800bec01ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:11.938 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058880, nodeCount=358, affectRows=358 2024-10-16 14:09:11.940 INFO 4486 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058880, nodeCount=358, validCount=358 2024-10-16 14:09:11.941 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058880, nodeCount=455, affectRows=455 2024-10-16 14:09:11.943 INFO 4486 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058880, nodeCount=455, validCount=455 2024-10-16 14:09:11.945 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058880, nodeCount=466, affectRows=466 2024-10-16 14:09:11.947 INFO 4486 --- [metric-archive-second-15,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058880, nodeCount=464, validCount=464 2024-10-16 14:09:11.948 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058880, nodeCount=549, affectRows=549 2024-10-16 14:09:11.949 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058880, nodeCount=622, affectRows=622 2024-10-16 14:09:11.950 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058880, nodeCount=549, validCount=549 2024-10-16 14:09:11.951 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058880, nodeCount=622, validCount=622 2024-10-16 14:09:11.995 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058880, nodeCount=1000, affectRows=1000 2024-10-16 14:09:11.999 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058880, nodeCount=999, validCount=999 2024-10-16 14:09:12.238 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,465c0da426d475bd,768e74ca15006215] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=465c0da426d475bd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:12.238 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,465c0da426d475bd,768e74ca15006215] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=465c0da426d475bd, duration=1 ms] 2024-10-16 14:09:12.520 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,02761d1b50c65ed3,f244c03f51ebcea4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=02761d1b50c65ed3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:12.521 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,02761d1b50c65ed3,f244c03f51ebcea4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=02761d1b50c65ed3, duration=1 ms] 2024-10-16 14:09:13.039 INFO 4486 --- [ocp-async-10,8abdcb9518965e25,dabb32ed9b1e84d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:09:13.666 INFO 4486 --- [ocp-async-9,928a48bd4054aa6e,888cf086b13551e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:13.679 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:13.994 INFO 4486 --- [prometheus-minute-schedule-1,4790a84e2319c921,73292ebecb440f5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:14.001 INFO 4486 --- [prometheus-minute-schedule-1,4790a84e2319c921,73292ebecb440f5a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:15.476 INFO 4486 --- [metric-archive-async-21,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058880, nodeCount=480, affectRows=480 2024-10-16 14:09:15.666 INFO 4486 --- [ocp-async-8,9bc5505588f976b0,f20d576b9162e495] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:16.278 INFO 4486 --- [metric-archive-async-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058880, nodeCount=480, validCount=480 2024-10-16 14:09:16.680 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:16.994 INFO 4486 --- [prometheus-minute-schedule-1,79b3c824741beb41,442d9914c03ca6a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:17.000 INFO 4486 --- [prometheus-minute-schedule-1,79b3c824741beb41,442d9914c03ca6a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:17.071 INFO 4486 --- [ocp-schedules-18,23846fe18dadd3e9,5c2370993ffac276] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:09:17.184 INFO 4486 --- [ocp-schedules-7,09f33a62a6afdf07,4e0772a7569f9cd4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:09:17.330 INFO 4486 --- [ocp-schedules-27,c12a00194fa0f80e,28a83a14134fb56a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:09:17.667 INFO 4486 --- [ocp-async-8,d97aac6aa052b6a1,0fb70d604d211b8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:18.686 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,cc513c22499e51b0,04ffb01c9ad695fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cc513c22499e51b0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:18.686 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,cc513c22499e51b0,04ffb01c9ad695fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cc513c22499e51b0, duration=3 ms] 2024-10-16 14:09:19.667 INFO 4486 --- [ocp-async-4,3bec27841b52bddd,e7db7b10e8809f88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:19.675 INFO 4486 --- [ocp-schedules-3,184f34138f7574d8,e5e9b816f16a007e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:09:19.681 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:19.994 INFO 4486 --- [prometheus-minute-schedule-1,03b955f1758367d2,ed73baf51a6909cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:20.001 INFO 4486 --- [prometheus-minute-schedule-1,03b955f1758367d2,ed73baf51a6909cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:20.074 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,02c4901e8814a1ac,a4f97606f9cf1ee3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=02c4901e8814a1ac, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:09:20.075 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,02c4901e8814a1ac,a4f97606f9cf1ee3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=02c4901e8814a1ac, duration=2 ms] 2024-10-16 14:09:20.299 INFO 4486 --- [ocp-schedules-16,e465af81f31f389e,95655a9c3e3b598c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:09:20.299 INFO 4486 --- [ocp-schedules-16,a2f46acf264aa153,58529a84d16596a0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:09:20.302 INFO 4486 --- [ocp-schedules-16,a2f46acf264aa153,58529a84d16596a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:09:20.333 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.334 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:20.403 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.404 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:20.427 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:20.428 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.428 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:09:20.429 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.430 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:20.458 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:20.459 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.460 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:20.461 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:09:20.461 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.464 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:09:20.465 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:20.473 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.474 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:20.484 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.492 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:20.493 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.493 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:20.494 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:09:20.498 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:09:20.502 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:20.503 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:20.521 INFO 4486 --- [ocp-schedules-30,684edbd7537fcd19,b5a230a6b7f8bdb5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:09:20.526 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:09:20.528 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:20.529 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.530 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:09:20.545 WARN 4486 --- [scanner-13339904-1,684edbd7537fcd19,1e5751609718e720] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:09:20.567 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:09:20.567 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:09:20.588 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:20.590 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.591 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:20.595 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.599 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:09:20.610 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:20.626 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:09:20.637 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:20.639 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.639 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:09:20.644 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.644 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:20.651 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:09:20.651 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058886919000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:09:20.679 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.680 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:20.710 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.710 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:20.738 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,692577cf2c54e186,967c55ca6c8cc0fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=692577cf2c54e186, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:20.739 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,692577cf2c54e186,967c55ca6c8cc0fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=692577cf2c54e186, duration=1 ms] 2024-10-16 14:09:20.741 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.741 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:20.765 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.765 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:20.795 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:20.822 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:20.823 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:20.824 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:20.829 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:09:20.833 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:09:20.838 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:09:20.839 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:09:20.869 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:09:20.996 INFO 4486 --- [prometheus-cache-monitor1,58e6b386725129b2,01ee702f59fca451] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13855,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3449,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8187} 2024-10-16 14:09:20.996 INFO 4486 --- [prometheus-cache-monitor1,58e6b386725129b2,01ee702f59fca451] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:09:21.015 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:09:21.040 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:21.042 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:21.043 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:09:21.070 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:09:21.071 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:09:21.564 INFO 4486 --- [ocp-schedules-15,6d0c7982809ef92e,72ab44e513b09c67] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:09:21.667 INFO 4486 --- [ocp-async-4,1244f74bb6facc97,ca6c701b6045f917] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:22.242 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,390410db57735661,fd0f930d981c855c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=390410db57735661, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:22.242 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,390410db57735661,fd0f930d981c855c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=390410db57735661, duration=1 ms] 2024-10-16 14:09:22.524 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b22e3663e265b4d5,3d6e7549d04a193a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b22e3663e265b4d5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:22.525 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,b22e3663e265b4d5,3d6e7549d04a193a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b22e3663e265b4d5, duration=1 ms] 2024-10-16 14:09:22.683 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:22.994 INFO 4486 --- [prometheus-minute-schedule-1,f52d53b219349164,62c8303582a9d8cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:23.000 INFO 4486 --- [prometheus-minute-schedule-1,f52d53b219349164,62c8303582a9d8cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:23.039 INFO 4486 --- [ocp-async-10,72e78f54f3acb0ec,8ef3ba85ec47b2cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:09:23.601 INFO 4486 --- [ocp-schedules-20,350cfa0a5405cc3d,50a516e467336fb6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:09:23.603 INFO 4486 --- [ocp-schedules-20,350cfa0a5405cc3d,50a516e467336fb6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:09:23.604 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:09:23.604 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:09:23.604 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:09:23.667 INFO 4486 --- [ocp-async-1,bd21749bfbc777f8,3faba96ab03f3e1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:24.262 INFO 4486 --- [ocp-schedules-29,68ab66c0b8857fad,1527ac9d25f8d12b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:09:25.321 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,419f303b59e42065,e2ae33796cbd8ee4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=419f303b59e42065, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:09:25.321 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,419f303b59e42065,e2ae33796cbd8ee4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=419f303b59e42065, duration=1 ms] 2024-10-16 14:09:25.476 INFO 4486 --- [ocp-schedules-25,4d81d4bbf5406b9f,eb9a5e7f9836f781] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:09:25.668 INFO 4486 --- [ocp-async-10,979faee48cd043e3,13f1fd3f84720971] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:25.684 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:25.994 INFO 4486 --- [prometheus-minute-schedule-1,e30bb35e86d3eb0f,60a6d9e4c74d9cf3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:26.002 INFO 4486 --- [prometheus-minute-schedule-1,e30bb35e86d3eb0f,60a6d9e4c74d9cf3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:27.360 INFO 4486 --- [ocp-schedules-18,d926122e6892b15b,4119ec495e6ea5fd] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:09:27.361 INFO 4486 --- [ocp-schedules-18,d926122e6892b15b,4119ec495e6ea5fd] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:09:27.391 INFO 4486 --- [Collect all tenants major compactions2,d926122e6892b15b,ff7709ef65d643cf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:09:27.391 INFO 4486 --- [Collect all tenants major compactions1,d926122e6892b15b,9718d39531b920cc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:09:27.391 INFO 4486 --- [Collect all tenants major compactions3,d926122e6892b15b,0891eee6648beef5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:09:27.392 INFO 4486 --- [Collect all tenants major compactions4,d926122e6892b15b,310e00c4c2297bdd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:09:27.392 INFO 4486 --- [Collect all tenants major compactions5,d926122e6892b15b,a202ff8cf27636bc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:09:27.403 INFO 4486 --- [Collect all tenants major compactions1,d926122e6892b15b,9718d39531b920cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:27.403 INFO 4486 --- [Collect all tenants major compactions2,d926122e6892b15b,ff7709ef65d643cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:27.404 INFO 4486 --- [Collect all tenants major compactions1,d926122e6892b15b,9718d39531b920cc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:09:27.405 INFO 4486 --- [Collect all tenants major compactions2,d926122e6892b15b,ff7709ef65d643cf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:09:27.408 INFO 4486 --- [Collect all tenants major compactions1,d926122e6892b15b,9718d39531b920cc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:09:27.408 INFO 4486 --- [Collect all tenants major compactions1,d926122e6892b15b,9718d39531b920cc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:09:27.408 INFO 4486 --- [Collect all tenants major compactions2,d926122e6892b15b,ff7709ef65d643cf] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:09:27.408 INFO 4486 --- [Collect all tenants major compactions2,d926122e6892b15b,ff7709ef65d643cf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:09:27.414 INFO 4486 --- [Collect all tenants major compactions3,d926122e6892b15b,0891eee6648beef5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:27.416 INFO 4486 --- [Collect all tenants major compactions3,d926122e6892b15b,0891eee6648beef5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:09:27.419 INFO 4486 --- [Collect all tenants major compactions3,d926122e6892b15b,0891eee6648beef5] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:09:27.420 INFO 4486 --- [Collect all tenants major compactions3,d926122e6892b15b,0891eee6648beef5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:09:27.422 INFO 4486 --- [Collect all tenants major compactions5,d926122e6892b15b,a202ff8cf27636bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:27.422 INFO 4486 --- [Collect all tenants major compactions4,d926122e6892b15b,310e00c4c2297bdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:27.423 INFO 4486 --- [Collect all tenants major compactions5,d926122e6892b15b,a202ff8cf27636bc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:09:27.423 INFO 4486 --- [Collect all tenants major compactions4,d926122e6892b15b,310e00c4c2297bdd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:09:27.427 INFO 4486 --- [Collect all tenants major compactions4,d926122e6892b15b,310e00c4c2297bdd] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:09:27.427 INFO 4486 --- [Collect all tenants major compactions4,d926122e6892b15b,310e00c4c2297bdd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:09:27.429 INFO 4486 --- [Collect all tenants major compactions5,d926122e6892b15b,a202ff8cf27636bc] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:09:27.429 INFO 4486 --- [Collect all tenants major compactions5,d926122e6892b15b,a202ff8cf27636bc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:09:27.430 INFO 4486 --- [ocp-schedules-18,d926122e6892b15b,4119ec495e6ea5fd] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:09:27.668 INFO 4486 --- [ocp-async-6,02210720a4fcdf79,14dad700add091ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:28.152 INFO 4486 --- [ocp-schedules-9,e4dc71ae7e9f9005,6d80724e821ea578] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:09:28.152+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:09:28.154 INFO 4486 --- [ocp-schedules-9,e4dc71ae7e9f9005,6d80724e821ea578] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:09:28.155 INFO 4486 --- [ocp-schedules-9,e4dc71ae7e9f9005,6d80724e821ea578] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:09:28.155 INFO 4486 --- [ocp-schedules-9,e4dc71ae7e9f9005,6d80724e821ea578] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:09:28.685 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:28.692 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,6095956a1424405b,46e79587c16600a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6095956a1424405b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:28.692 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,6095956a1424405b,46e79587c16600a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6095956a1424405b, duration=1 ms] 2024-10-16 14:09:28.994 INFO 4486 --- [prometheus-minute-schedule-1,27c8093ec31a02dd,012305db098b6181] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:29.000 INFO 4486 --- [prometheus-minute-schedule-1,27c8093ec31a02dd,012305db098b6181] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:29.668 INFO 4486 --- [ocp-async-7,02722b703218d052,8ca1e50627df4bd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:30.104 INFO 4486 --- [prometheus-second-schedule-1,e86e95982da7a490,37b851484eecbadf] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:09:30.259 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d4ebf257d0697e63,0aea774706c89395] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=d4ebf257d0697e63, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:09:30.259 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,d4ebf257d0697e63,0aea774706c89395] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=d4ebf257d0697e63, duration=1 ms] 2024-10-16 14:09:30.675 INFO 4486 --- [ocp-schedules-14,0e09921a288a5e09,65ef2f23bbefbbf6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:09:30.699 INFO 4486 --- [ocp-schedules-19,bd9aea81cda80200,3744961d100ce854] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=3, successCount=3 2024-10-16 14:09:30.712 WARN 4486 --- [scanner-13340142-1,0e09921a288a5e09,9cde07fbac5b4e84] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:09:30.754 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,65149468f2b085ad,5e9330c8deb00777] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=65149468f2b085ad, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:30.755 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,65149468f2b085ad,5e9330c8deb00777] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=65149468f2b085ad, duration=9 ms] 2024-10-16 14:09:30.770 INFO 4486 --- [alarm-detect-14,0e09921a288a5e09,e2021783218c4afe] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058940 ~ 1729058960 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058955, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058955, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058955=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:09:31.571 INFO 4486 --- [ocp-schedules-6,82627b3da981197d,d8ea7399c43448df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:09:31.668 INFO 4486 --- [ocp-async-8,af79b06c6fa825b2,57450f74792102c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:31.685 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:31.994 INFO 4486 --- [prometheus-minute-schedule-1,ec68b125d6fd0c0f,d907b41e150c8f57] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:32.001 INFO 4486 --- [prometheus-minute-schedule-1,ec68b125d6fd0c0f,d907b41e150c8f57] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:32.247 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,44bcfaf60ca9fd86,c21578e3d31fbf91] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=44bcfaf60ca9fd86, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:32.247 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,44bcfaf60ca9fd86,c21578e3d31fbf91] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=44bcfaf60ca9fd86, duration=1 ms] 2024-10-16 14:09:32.530 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,60278f8a835b40ed,b3f4fe5135644aae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=60278f8a835b40ed, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:32.530 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,60278f8a835b40ed,b3f4fe5135644aae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=60278f8a835b40ed, duration=1 ms] 2024-10-16 14:09:33.039 INFO 4486 --- [ocp-async-3,fea21de6401cf9b5,b2877262619f6e28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:09:33.605 INFO 4486 --- [ocp-schedules-20,350cfa0a5405cc3d,50a516e467336fb6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:09:33.668 INFO 4486 --- [ocp-async-2,288c792c0f2cec74,4964aee83319bf22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:34.686 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:34.994 INFO 4486 --- [prometheus-minute-schedule-1,c6dc6af2d075eeae,0b99a6561aaccf9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:35.004 INFO 4486 --- [prometheus-minute-schedule-1,c6dc6af2d075eeae,0b99a6561aaccf9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:35.669 INFO 4486 --- [ocp-async-9,e68462db6079e1c4,5a51d05fd37b07e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:37.063 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:09:37.064 INFO 4486 --- [ocp-schedules-26,98806c4a6e041548,3b194c4473b5f468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:09:37.066 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:09:37.067 INFO 4486 --- [ocp-schedules-29,118ce7b15d3290bf,05d51399f224a134] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:09:37.068 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:09:37.069 INFO 4486 --- [ocp-schedules-29,118ce7b15d3290bf,05d51399f224a134] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:09:37.070 INFO 4486 --- [ocp-schedules-26,98806c4a6e041548,3b194c4473b5f468] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:09:37.081 INFO 4486 --- [ocp-schedules-1,ebc3e232f321fb07,a064d54f5be587ff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:09:37.083 INFO 4486 --- [ocp-schedules-1,ebc3e232f321fb07,a064d54f5be587ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:09:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:09:37.090 INFO 4486 --- [ocp-schedules-2,286332715ea1fe14,2252a0d1f4af7761] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:09:37.091 INFO 4486 --- [ocp-schedules-9,2cba738e0fa533d5,5d6fdcd98a4125d2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:09:37.091 INFO 4486 --- [ocp-schedules-10,ac43281b7713e086,ca8f3b830bcce410] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:09:37.091 INFO 4486 --- [host-check-agent-status1,286332715ea1fe14,1bf5594260ce23e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:09:37.091 INFO 4486 --- [host-check-agent-status2,286332715ea1fe14,f7dcde4bdeba0446] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:09:37.091 INFO 4486 --- [host-check-agent-status3,286332715ea1fe14,563ae777702931f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:09:37.091 INFO 4486 --- [host-check-agent-status4,286332715ea1fe14,2e652aa8b3748713] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:09:37.092 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:09:37.092 INFO 4486 --- [host-check-clockdiff1,2cba738e0fa533d5,f8f217325800670c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:09:37.093 INFO 4486 --- [host-check-clockdiff2,2cba738e0fa533d5,13e48175e69c1d52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:09:37.093 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:09:37.093 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:09:37.094 INFO 4486 --- [host-check-clockdiff3,2cba738e0fa533d5,a7882a2fb83d13ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:09:37.094 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:09:37.094 INFO 4486 --- [host-check-clockdiff4,2cba738e0fa533d5,1df6ae889ab93ab3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:09:37.100 INFO 4486 --- [host-check-agent-status1,286332715ea1fe14,1bf5594260ce23e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:37.100 INFO 4486 --- [host-check-agent-status3,286332715ea1fe14,563ae777702931f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:09:37.100 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:09:37.101 INFO 4486 --- [host-check-agent-status2,286332715ea1fe14,f7dcde4bdeba0446] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:09:37.101 INFO 4486 --- [host-check-agent-status4,286332715ea1fe14,2e652aa8b3748713] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:09:37.101 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:37.102 INFO 4486 --- [host-check-agent-status1,286332715ea1fe14,1bf5594260ce23e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:37.102 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:09:37.102 INFO 4486 --- [host-check-agent-status3,286332715ea1fe14,563ae777702931f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:09:37.136 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:09:37.137 INFO 4486 --- [host-check-agent-status2,286332715ea1fe14,f7dcde4bdeba0446] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:09:37.137 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:37.137 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:09:37.140 INFO 4486 --- [host-check-agent-status4,286332715ea1fe14,2e652aa8b3748713] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:09:37.147 INFO 4486 --- [host-check-clockdiff4,2cba738e0fa533d5,1df6ae889ab93ab3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058977 4 4 , error= 2024-10-16 14:09:37.151 INFO 4486 --- [host-check-clockdiff1,2cba738e0fa533d5,f8f217325800670c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058977 0 0 , error= 2024-10-16 14:09:37.152 INFO 4486 --- [host-check-clockdiff2,2cba738e0fa533d5,13e48175e69c1d52] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058977 4 4 , error= 2024-10-16 14:09:37.154 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:09:37.154 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:09:37.169 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.170 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:37.172 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:09:37.232 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:37.233 INFO 4486 --- [host-check-agent-status3,286332715ea1fe14,563ae777702931f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:09:37.233 INFO 4486 --- [host-check-agent-status1,286332715ea1fe14,1bf5594260ce23e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:09:37.234 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:09:37.240 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:09:37.240 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:09:37.240 INFO 4486 --- [host-check-agent-status2,286332715ea1fe14,f7dcde4bdeba0446] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:09:37.241 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.241 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:09:37.240633172+08:00 2024-10-16 14:09:37.242 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:09:37.242 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:09:37.245951728+08:00 2024-10-16 14:09:37.242 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:09:37.241+08:00 2024-10-16 14:09:37.242 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:09:37.242+08:00 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:09:37.247009474+08:00 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-agent-status3,286332715ea1fe14,563ae777702931f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-agent-status3,286332715ea1fe14,563ae777702931f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-agent-status3,286332715ea1fe14,563ae777702931f7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:09:37.243 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-agent-status1,286332715ea1fe14,1bf5594260ce23e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-agent-status1,286332715ea1fe14,1bf5594260ce23e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-agent-status1,286332715ea1fe14,1bf5594260ce23e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:09:37.243 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:09:37.243+08:00 2024-10-16 14:09:37.244 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:37.244 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.244 INFO 4486 --- [host-check-agent-status4,286332715ea1fe14,2e652aa8b3748713] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:09:37.245 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.245 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:09:37.241+08:00 and status AVAILABLE 2024-10-16 14:09:37.245 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:09:37.242+08:00 and status AVAILABLE 2024-10-16 14:09:37.245 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:09:37.246 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:09:37.243+08:00 and status AVAILABLE 2024-10-16 14:09:37.246 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:09:37.247 INFO 4486 --- [host-check-agent-status2,286332715ea1fe14,f7dcde4bdeba0446] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:09:37.247 INFO 4486 --- [host-check-agent-status2,286332715ea1fe14,f7dcde4bdeba0446] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:09:37.247 INFO 4486 --- [host-check-agent-status2,286332715ea1fe14,f7dcde4bdeba0446] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:09:37.247 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:09:37.247 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:09:37.251469761+08:00 2024-10-16 14:09:37.247 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:09:37.248 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:09:37.248 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:09:37.247+08:00 2024-10-16 14:09:37.249 INFO 4486 --- [host-check-agent-status4,286332715ea1fe14,2e652aa8b3748713] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:09:37.249 INFO 4486 --- [host-check-agent-status4,286332715ea1fe14,2e652aa8b3748713] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:09:37.249 INFO 4486 --- [host-check-agent-status4,286332715ea1fe14,2e652aa8b3748713] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:09:37.249 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:09:37.251 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:09:37.247+08:00 and status AVAILABLE 2024-10-16 14:09:37.252 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:09:37.255 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:37.256 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:09:37.256 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:09:37.256 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.257 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:37.257 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:09:37.257 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:09:37.262 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:09:37.263 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:09:37.263 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:09:37.265 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.265 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.265 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.265 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.265 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.265 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.271 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.272 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:09:37.279 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:37.282 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:37.282 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:09:37.284 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.285 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:09:37.286 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:37.289 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:09:37.297 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.298 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:09:37.301 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.303 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:09:37.308 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:09:37.314 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.315 INFO 4486 --- [ocp-schedules-24,0bd1dd604f5232d8,3532416a8d3d95a6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:09:37.317 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:09:37.318 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:09:37.324 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:09:37.328 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:09:37.334 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:09:37.334 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:37.345 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:37.347 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:09:37.347 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:37.357 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:09:37.363 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:09:37.372 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:09:37.373 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:09:37.386 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:09:37.387 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:09:37.388 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:37.396 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:09:37.397 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:09:37.413 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:37.414 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:37.420 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:09:37.428 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:09:37.430 INFO 4486 --- [host-check-host-status3,ac43281b7713e086,505b51545653847b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:09:37.439 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:09:37.447 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:09:37.455 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:09:37.457 INFO 4486 --- [host-check-host-status4,ac43281b7713e086,ddc1b0a50c6239f7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:09:37.458 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:09:37.469 INFO 4486 --- [host-check-host-status2,ac43281b7713e086,3518453ed60da214] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:09:37.509 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:37.511 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.512 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:09:37.514 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.514 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.516 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:09:37.518 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:09:37.528 INFO 4486 --- [host-check-host-status1,ac43281b7713e086,58fdb02884b17c2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:09:37.529 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.529 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.529 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.529 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.529 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.530 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.565 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:09:37.616 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:09:37.665 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:37.669 INFO 4486 --- [ocp-async-10,b00453871fc32d63,0c23892d0318712e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:37.683 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:37.684 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.685 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:09:37.686 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:09:37.687 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.687 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:37.688 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:09:37.689 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.689 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.690 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:09:37.699 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.699 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.699 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.699 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.699 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.700 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.707 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:09:37.902 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:09:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:09:37.969 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:37.971 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.972 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:09:37.976 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.976 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.979 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:09:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:09:37.993 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.993 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.993 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.993 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.993 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:37.993 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:37.994 INFO 4486 --- [prometheus-minute-schedule-1,9d037b353b5ee7c2,22ad70267dff8afc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:38.000 INFO 4486 --- [prometheus-minute-schedule-1,9d037b353b5ee7c2,22ad70267dff8afc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:38.011 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:09:38.060 INFO 4486 --- [host-check-clockdiff3,2cba738e0fa533d5,a7882a2fb83d13ea] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729058978 4 4 , error= 2024-10-16 14:09:38.237 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:09:38.283 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:38.285 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:38.287 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:09:38.288 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:38.288 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:38.290 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:09:38.299 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:38.299 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:38.299 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:38.299 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:38.299 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:09:38.300 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:38.362 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:09:38.485 INFO 4486 --- [ocp-schedules-2,11b4d2e7700958b5,bfd6c53d9f6d1c35] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13855,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8187} 2024-10-16 14:09:38.485 INFO 4486 --- [ocp-schedules-2,11b4d2e7700958b5,bfd6c53d9f6d1c35] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:09:38.567 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:09:38.366819, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:09:38.568 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:09:38.366819, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:09:38.606 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:09:38.366819} } 2024-10-16 14:09:38.661 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:09:38.669 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:09:38.672 INFO 4486 --- [ocp-schedules-17,d2583e1de11a39c2,c9dd1a83b2765a2c] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:09:38.697 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,cfb763643df56133,ce396221fe256ff2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=cfb763643df56133, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:38.698 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,cfb763643df56133,ce396221fe256ff2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=cfb763643df56133, duration=2 ms] 2024-10-16 14:09:39.669 INFO 4486 --- [ocp-async-2,2f81bc0e40cb5ae2,217b582341661229] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:39.680 INFO 4486 --- [ocp-schedules-26,add660d8ca11bd96,2d003b2acb5b21b3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:09:40.006 INFO 4486 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:09:40.006 INFO 4486 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:09:40.026 INFO 4486 --- [Check obproxy2,ff4e72f27f686867,4e5329f3bdde820f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:09:40.026 INFO 4486 --- [Check obproxy2,ff4e72f27f686867,4e5329f3bdde820f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:40.027 INFO 4486 --- [Check obproxy1,2b4d725a8b6247f6,644cf5f556a8b1fe] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:09:40.027 INFO 4486 --- [Check obproxy1,2b4d725a8b6247f6,644cf5f556a8b1fe] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:40.032 INFO 4486 --- [Collect obproxy2,96572cfdb90cf2f0,a6bb72c0d51b84ff] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:09:40.032 INFO 4486 --- [Collect obproxy2,96572cfdb90cf2f0,a6bb72c0d51b84ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:40.032 INFO 4486 --- [Collect obproxy1,f15dd4112087d1bc,5851d1f627404bf0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:09:40.032 INFO 4486 --- [Collect obproxy1,f15dd4112087d1bc,5851d1f627404bf0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:09:40.049 INFO 4486 --- [Check obproxy1,2b4d725a8b6247f6,644cf5f556a8b1fe] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:09:40.049 INFO 4486 --- [Check obproxy1,2b4d725a8b6247f6,644cf5f556a8b1fe] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:09:40.051 INFO 4486 --- [Check obproxy2,ff4e72f27f686867,4e5329f3bdde820f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:09:40.051 INFO 4486 --- [Check obproxy2,ff4e72f27f686867,4e5329f3bdde820f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:09:40.054 INFO 4486 --- [Collect obproxy2,96572cfdb90cf2f0,a6bb72c0d51b84ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:09:40.054 INFO 4486 --- [Collect obproxy1,f15dd4112087d1bc,5851d1f627404bf0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:09:40.057 INFO 4486 --- [Collect obproxy1,f15dd4112087d1bc,5851d1f627404bf0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:09:40.060 INFO 4486 --- [Collect obproxy2,96572cfdb90cf2f0,a6bb72c0d51b84ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:09:40.060 INFO 4486 --- [Collect obproxy1,f15dd4112087d1bc,5851d1f627404bf0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:09:40.065 INFO 4486 --- [Collect obproxy2,96572cfdb90cf2f0,a6bb72c0d51b84ff] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:09:40.688 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:40.761 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eba1443dfc640bf2,1c47454a8494f32a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eba1443dfc640bf2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:40.761 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,eba1443dfc640bf2,1c47454a8494f32a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eba1443dfc640bf2, duration=1 ms] 2024-10-16 14:09:40.857 INFO 4486 --- [ocp-schedules-20,76f44d39aa0bdccd,ea76dcc96199d6c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:09:40.888 WARN 4486 --- [scanner-13340464-1,76f44d39aa0bdccd,01e40df93f86c72c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:09:40.960 INFO 4486 --- [alarm-detect-19,76f44d39aa0bdccd,0011f14ed7b7a373] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058955 ~ 1729058970 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058970, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058970, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058970=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:09:40.996 INFO 4486 --- [prometheus-hour-schedule-1,e7ec4a1c415a70a1,56e5270ff7931c8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:09:40.996 INFO 4486 --- [prometheus-minute-schedule-1,3382593584ac49d9,8ffbf17bca0fae0a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:41.013 INFO 4486 --- [prometheus-minute-schedule-1,3382593584ac49d9,8ffbf17bca0fae0a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:41.577 INFO 4486 --- [ocp-schedules-27,eec385526c93502c,23fb33b3fab352d7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:09:41.669 INFO 4486 --- [ocp-async-9,18c0d2da46e4bb91,e5433197b2b151c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:42.212 INFO 4486 --- [ocp-schedules-5,68bbee3e4ffb8b3c,9ebe31b936facfb3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:09:42.252 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a60ad73decb7fe5b,15018557eb7ca763] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=a60ad73decb7fe5b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:42.253 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,a60ad73decb7fe5b,15018557eb7ca763] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=a60ad73decb7fe5b, duration=2 ms] 2024-10-16 14:09:42.535 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,75fba2c8f409c76d,03b5846040ba64e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=75fba2c8f409c76d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:42.536 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,75fba2c8f409c76d,03b5846040ba64e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=75fba2c8f409c76d, duration=2 ms] 2024-10-16 14:09:43.040 INFO 4486 --- [ocp-async-4,ae7a9ec2a6f5a4ee,66f72eac65661d90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:09:43.670 INFO 4486 --- [ocp-async-8,3b38f46add3d031b,ab403a6000ec58bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:43.689 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:43.994 INFO 4486 --- [prometheus-minute-schedule-1,84a08b76090cc493,ae09af12c72326f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:44.002 INFO 4486 --- [prometheus-minute-schedule-1,84a08b76090cc493,ae09af12c72326f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:45.658 INFO 4486 --- [ocp-schedules-27,53f0c2d854963cc0,e00dc4d8e2a11268] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:09:45.664 INFO 4486 --- [ocp-schedules-27,53f0c2d854963cc0,e00dc4d8e2a11268] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:09:45.664 INFO 4486 --- [ocp-schedules-27,53f0c2d854963cc0,e00dc4d8e2a11268] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:09:45.670 INFO 4486 --- [ocp-async-7,fc867c2cd6fea42b,847ebd920fa609d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:45.676 INFO 4486 --- [ocp-schedules-19,770c751004d9636d,679482c33f20b990] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:09:45.676 INFO 4486 --- [ocp-schedules-19,770c751004d9636d,679482c33f20b990] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:09:46.690 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:46.994 INFO 4486 --- [prometheus-minute-schedule-1,0fa83bd195540d9a,1825931b5adc3d3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:47.000 INFO 4486 --- [prometheus-minute-schedule-1,0fa83bd195540d9a,1825931b5adc3d3c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:47.070 INFO 4486 --- [ocp-schedules-29,118ce7b15d3290bf,05d51399f224a134] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:09:47.250 INFO 4486 --- [ocp-schedules-1,ebc3e232f321fb07,a064d54f5be587ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:09:47.333 INFO 4486 --- [ocp-schedules-15,d82774de036fe67f,60a7477691cb3003] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:09:47.670 INFO 4486 --- [ocp-async-2,628fe042ab5846f8,89a1c027d7311c8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:48.703 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e6c0878be45e3d8d,aedd52bc538e73ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=e6c0878be45e3d8d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:48.703 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,e6c0878be45e3d8d,aedd52bc538e73ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=e6c0878be45e3d8d, duration=2 ms] 2024-10-16 14:09:48.785 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:48.786 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:09:48.788 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:48.788 INFO 4486 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:09:48.789 INFO 4486 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Follower checked 2024-10-16 14:09:49.670 INFO 4486 --- [ocp-async-1,ab976089ffd65c1d,afd8b05b6a120ee0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:49.690 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:49.994 INFO 4486 --- [prometheus-minute-schedule-1,9a0057eac97dc1ee,ac6e6926e2959da1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:50.000 INFO 4486 --- [prometheus-minute-schedule-1,9a0057eac97dc1ee,ac6e6926e2959da1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:50.625 INFO 4486 --- [ocp-schedules-7,bf329abc41c04a50,4aabe8da96ca9909] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:09:50.625 INFO 4486 --- [ocp-schedules-7,bf329abc41c04a50,4aabe8da96ca9909] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:09:50.627 INFO 4486 --- [ocp-schedules-7,bf329abc41c04a50,4aabe8da96ca9909] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:09:50.627 INFO 4486 --- [ocp-schedules-7,bf329abc41c04a50,4aabe8da96ca9909] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:09:50.627 INFO 4486 --- [ocp-schedules-7,bf329abc41c04a50,4aabe8da96ca9909] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:09:50.627 INFO 4486 --- [ocp-schedules-7,bf329abc41c04a50,4aabe8da96ca9909] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703046246634950656 2024-10-16 14:09:50.766 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,200aa5da91139f3d,b0a1221388b038dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=200aa5da91139f3d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:50.766 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,200aa5da91139f3d,b0a1221388b038dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=200aa5da91139f3d, duration=1 ms] 2024-10-16 14:09:50.957 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2024-10-16 14:09:50.957 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2024-10-16 14:09:50.968 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2024-10-16 14:09:50.968 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2024-10-16 14:09:50.968 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2024-10-16 14:09:50.977 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2024-10-16 14:09:50.977 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2024-10-16 14:09:50.977 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2024-10-16 14:09:50.983 INFO 4486 --- [ocp-schedules-17,7c4a4139e6e36c2c,0de95675bea903f7] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2024-10-16 14:09:51.025 INFO 4486 --- [ocp-schedules-19,8154aa583eb9edd6,ac7e1d8db5d63eb3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:09:51.065 WARN 4486 --- [scanner-13340704-1,8154aa583eb9edd6,6c8515bed9a9ca6d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:09:51.084 INFO 4486 --- [ocp-schedules-16,a2f46acf264aa153,58529a84d16596a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:09:51.084 INFO 4486 --- [ocp-schedules-16,e1537adbd98f2557,d820f89b4958229d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:09:51.087 INFO 4486 --- [ocp-schedules-16,e1537adbd98f2557,d820f89b4958229d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:09:51.133 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.134 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:51.142 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.150 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:51.180 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.181 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:51.185 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.186 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.186 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:09:51.215 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.216 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.217 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:51.218 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:09:51.218 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.221 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:09:51.222 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:51.231 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.231 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:51.241 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.251 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.252 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.253 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:51.254 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:09:51.259 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:09:51.263 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:51.265 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:51.282 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.284 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.285 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:09:51.290 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:09:51.317 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:09:51.317 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:09:51.330 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.332 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.333 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:09:51.336 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.340 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:09:51.343 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:51.369 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:09:51.370 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.370 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:51.380 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.381 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.382 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:09:51.394 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:09:51.394 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058886919000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:09:51.403 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.404 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:51.432 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.433 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:51.459 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.459 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:09:51.485 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.485 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:09:51.513 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:09:51.543 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.545 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.546 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:09:51.551 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:09:51.555 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:09:51.560 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:09:51.562 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:09:51.583 INFO 4486 --- [ocp-schedules-27,702b77063a72fdbb,e75e6659277ac5f6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:09:51.584 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:09:51.671 INFO 4486 --- [ocp-async-8,f9d41875c74c9b0a,a9e2d10ca9a2ea72] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:51.728 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:09:51.760 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:09:51.761 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:09:51.763 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:09:51.793 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:09:51.793 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:09:51.957 INFO 4486 --- [ocp-schedules-20,7e2354a7ae0242b2,06f2f26e309c518c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 10.197.49.171, 10.197.49.173, 172.28.46.152] 2024-10-16 14:09:51.958 INFO 4486 --- [ocp-schedules-20,7e2354a7ae0242b2,06f2f26e309c518c] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000001 2024-10-16 14:09:51.958 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000015 2024-10-16 14:09:51.958 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2024-10-16 14:09:51.958 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2024-10-16 14:09:51.958 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=7 2024-10-16 14:09:51.958 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1000003 2024-10-16 14:09:51.963 INFO 4486 --- [ocp-schedules-20,7e2354a7ae0242b2,06f2f26e309c518c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000001 2024-10-16 14:09:51.963 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:09:51.964 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:09:51.964 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:09:51.965 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:09:51.965 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000003 2024-10-16 14:09:51.965 INFO 4486 --- [ocp-schedules-20,7e2354a7ae0242b2,06f2f26e309c518c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000001 2024-10-16 14:09:51.965 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:09:51.965 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:09:51.965 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:09:51.966 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:09:51.966 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000003 2024-10-16 14:09:51.988 INFO 4486 --- [ForkJoinPool.commonPool-worker-30,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62889/api/v1/log/alarms, params:null 2024-10-16 14:09:51.989 INFO 4486 --- [ForkJoinPool.commonPool-worker-8,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62889/api/v1/log/alarms, params:null 2024-10-16 14:09:51.990 INFO 4486 --- [ForkJoinPool.commonPool-worker-31,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62889/api/v1/log/alarms, params:null 2024-10-16 14:09:51.991 INFO 4486 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.173:62889/api/v1/log/alarms, params:null 2024-10-16 14:09:51.992 INFO 4486 --- [ForkJoinPool.commonPool-worker-19,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62889/api/v1/log/alarms, params:null 2024-10-16 14:09:51.992 INFO 4486 --- [ocp-schedules-20,7e2354a7ae0242b2,06f2f26e309c518c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.171:62889/api/v1/log/alarms, params:null 2024-10-16 14:09:52.002 INFO 4486 --- [ocp-schedules-20,7e2354a7ae0242b2,06f2f26e309c518c] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2024-10-16 14:09:52.257 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,279273b91edeb471,c72b5e0916f70523] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=279273b91edeb471, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:52.257 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-7,279273b91edeb471,c72b5e0916f70523] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=279273b91edeb471, duration=1 ms] 2024-10-16 14:09:52.544 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,b714b5fa4f6136a2,ed6c63357d5018f2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b714b5fa4f6136a2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:52.544 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-8,b714b5fa4f6136a2,ed6c63357d5018f2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b714b5fa4f6136a2, duration=2 ms] 2024-10-16 14:09:52.691 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:52.994 INFO 4486 --- [prometheus-minute-schedule-1,5f80c1ff56648e02,316ac745b9d2286d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:52.999 INFO 4486 --- [prometheus-minute-schedule-1,5f80c1ff56648e02,316ac745b9d2286d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:53.040 INFO 4486 --- [ocp-async-6,8f5447d4d2d98297,ca2b2d941716572d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:09:53.671 INFO 4486 --- [ocp-async-5,e7c5c72f880f6e01,f1821ac5d0615481] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:55.671 INFO 4486 --- [ocp-async-6,451b35314d181eb6,a9db96bf9319fb70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:55.692 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:55.994 INFO 4486 --- [prometheus-minute-schedule-1,ecb78d995b03cf7c,835262a49d2993ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:56.001 INFO 4486 --- [prometheus-minute-schedule-1,ecb78d995b03cf7c,835262a49d2993ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:57.671 INFO 4486 --- [ocp-async-7,4635b31d8071765e,2f79eb33b3ac9f82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:58.693 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:09:58.709 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,fcdedd57099d0f53,c8d1e2e232efdab4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fcdedd57099d0f53, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:09:58.710 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-1,fcdedd57099d0f53,c8d1e2e232efdab4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fcdedd57099d0f53, duration=2 ms] 2024-10-16 14:09:58.994 INFO 4486 --- [prometheus-minute-schedule-1,bdfae5fa1fd9e416,2842fd33008e2fc4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059000, interval=60 2024-10-16 14:09:59.001 INFO 4486 --- [prometheus-minute-schedule-1,bdfae5fa1fd9e416,2842fd33008e2fc4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:09:59.672 INFO 4486 --- [ocp-async-2,ba25bcdf4cafd478,dda9b5d81dc389e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:09:59.685 INFO 4486 --- [ocp-schedules-3,cb6797db7020e90c,da59a3152b91841e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:10:00.235 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 14:05:00.0, nextRun=2024-10-16 14:10:00.0, currentDate=Wed Oct 16 14:10:00 CST 2024 2024-10-16 14:10:00.235 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 14:05:00.0, nextRunTime=2024-10-16 14:10:00.0} 2024-10-16 14:10:00.244 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2024-10-16 14:10:00.245 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.197.49.184', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-10-16T14:10:00.245+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='791e68b70ade5cad', operation=EXECUTE} 2024-10-16 14:10:00.267 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 14:05:01.0, nextRun=2024-10-16 14:10:00.0, currentDate=Wed Oct 16 14:10:00 CST 2024 2024-10-16 14:10:00.267 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2024-10-16 14:05:01.0, nextRunTime=2024-10-16 14:10:00.0} 2024-10-16 14:10:00.298 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 1000003] 2024-10-16 14:10:00.348 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.p.s.internal.SqlParseTaskSchedule : The last sql parse service is still running, skip current task 2024-10-16 14:10:00.348 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,ef8097c52c7534c4] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Task is not ready for schedule, taskName=SqlParseSchedule 2024-10-16 14:10:00.770 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,54093624f7859540,07c71e326389eb49] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=54093624f7859540, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:00.770 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-5,54093624f7859540,07c71e326389eb49] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=54093624f7859540, duration=1 ms] 2024-10-16 14:10:01.203 INFO 4486 --- [ocp-schedules-9,e2793829c6daa0a8,903ee967bff0915e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:10:01.242 WARN 4486 --- [scanner-13341012-1,e2793829c6daa0a8,70c125d182bd1eed] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:10:01.300 INFO 4486 --- [alarm-detect-11,e2793829c6daa0a8,cf2d5653980d1b9f] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058970 ~ 1729058991 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058985, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729058985, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729058985=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:10:01.429 INFO 4486 --- [alarm-detect-11,e2793829c6daa0a8,cf2d5653980d1b9f] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=host_ntp_service_not_exist, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.197.49.176-2] 2024-10-16 14:10:01.472 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,020d2339a7dc820d,faed33b45764fd2c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=020d2339a7dc820d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:10:01.473 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-9,020d2339a7dc820d,faed33b45764fd2c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=020d2339a7dc820d, duration=7 ms] 2024-10-16 14:10:01.588 INFO 4486 --- [ocp-schedules-7,981fdd541480ac40,9a06c4b8bfce7ae9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:10:01.672 INFO 4486 --- [ocp-async-2,00c57f15919a07d5,1b38d4ec59081a16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:01.694 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:01.961 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,224c819afb93679b,5f96618059b59ce2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms?isSubscribedByMe=false&status=Active&page=1&size=10, client=10.197.49.40, traceId=224c819afb93679b, method=ResponseEntity com.oceanbase.ocp.server.common.controller.AlarmController.fetchAlarms(Integer,Integer,String,String,List,String,List,List,OffsetDateTime,OffsetDateTime,String), args=1,10,[Active],] 2024-10-16 14:10:01.994 INFO 4486 --- [prometheus-minute-schedule-1,2b60e23ffe3671a4,3e26a210893f8b6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:02.004 INFO 4486 --- [prometheus-minute-schedule-1,2b60e23ffe3671a4,3e26a210893f8b6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:02.215 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,224c819afb93679b,5f96618059b59ce2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms client=10.197.49.40, traceId=224c819afb93679b, duration=273 ms] 2024-10-16 14:10:02.262 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,838fc7a3739eb971,db0b81f584552f6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=838fc7a3739eb971, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:02.263 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,838fc7a3739eb971,db0b81f584552f6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=838fc7a3739eb971, duration=2 ms] 2024-10-16 14:10:02.551 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1adfe0e8624799dc,9028bef7a7800714] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1adfe0e8624799dc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:02.551 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,1adfe0e8624799dc,9028bef7a7800714] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1adfe0e8624799dc, duration=1 ms] 2024-10-16 14:10:03.040 INFO 4486 --- [ocp-async-7,5bb14d65e03f52ea,d96f3c51f49dd072] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:10:03.186 INFO 4486 --- [ocp-schedules-29,3a2e95a69013b76e,7f50f49f3a02fa8b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2024-10-16T14:10:03.186+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:10:03.190 INFO 4486 --- [ocp-schedules-29,3a2e95a69013b76e,7f50f49f3a02fa8b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster OceanBase for slowSqlStat, sql count is 0. 2024-10-16 14:10:03.202 INFO 4486 --- [ocp-schedules-29,3a2e95a69013b76e,7f50f49f3a02fa8b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster obce_3zones, exclude result is empty. 2024-10-16 14:10:03.213 INFO 4486 --- [ocp-schedules-29,3a2e95a69013b76e,7f50f49f3a02fa8b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster glob_3zone, exclude result is empty. 2024-10-16 14:10:03.672 INFO 4486 --- [ocp-async-8,c553f7aa893e2841,87d516870edee3e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:04.493 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,603d29a75d38e1c6] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2024-10-16 14:00:05.0, nextRun=2024-10-16 14:10:04.0, currentDate=Wed Oct 16 14:10:04 CST 2024 2024-10-16 14:10:04.493 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,603d29a75d38e1c6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2024-10-16 14:00:05.0, nextRunTime=2024-10-16 14:10:04.0} 2024-10-16 14:10:04.496 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,603d29a75d38e1c6] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2024-10-16 14:10:04.496 INFO 4486 --- [task-coordinator1,791e68b70ade5cad,603d29a75d38e1c6] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.197.49.184', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2024-10-16T14:10:04.496+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='791e68b70ade5cad', operation=EXECUTE} 2024-10-16 14:10:04.695 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:04.994 INFO 4486 --- [prometheus-minute-schedule-1,a715a496372bdf85,62369b883be03f05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:05.000 INFO 4486 --- [prometheus-minute-schedule-1,a715a496372bdf85,62369b883be03f05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:05.241 INFO 4486 --- [subtask-coordinator2,791e68b70ade5cad,23b57ba7a81314e2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934617, operation:EXECUTE, state: RUNNING 2024-10-16 14:10:05.266 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=7934617, context=Context{parallelIdx=-1, stringMap={task_instance_id=5266501, task_operation=execute, latest_execution_start_time=2024-10-16T14:10:05.227+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=7934617}, listMap={}}, executor=10.197.49.184 2024-10-16 14:10:05.271 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2024-10-16 14:10:05.272 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2024-10-16 14:10:05.273 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.274 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:10:05.275 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.276 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.276 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.277 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2024-10-16 14:10:05.294 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.295 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.295 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.296 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2024-10-16 14:10:05.310 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.311 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.312 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.312 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2024-10-16 14:10:05.328 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.328 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.329 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.330 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2024-10-16 14:10:05.344 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.344 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.345 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.346 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2024-10-16 14:10:05.353 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.353 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.354 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.354 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2024-10-16 14:10:05.359 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.360 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.361 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.361 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2024-10-16 14:10:05.367 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.367 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.368 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.368 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2024-10-16 14:10:05.375 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.375 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.376 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.377 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2024-10-16 14:10:05.383 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.383 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.384 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.385 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2024-10-16 14:10:05.391 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.392 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.392 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.393 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2024-10-16 14:10:05.402 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.403 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.403 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.404 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2024-10-16 14:10:05.418 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.419 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.420 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.420 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2024-10-16 14:10:05.426 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.426 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.427 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.428 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2024-10-16 14:10:05.436 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.436 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.437 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.437 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2024-10-16 14:10:05.444 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.445 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.446 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.447 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2024-10-16 14:10:05.453 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.453 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.454 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.455 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2024-10-16 14:10:05.461 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.462 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.463 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.463 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2024-10-16 14:10:05.469 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.470 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.471 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.472 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2024-10-16 14:10:05.478 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.479 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.480 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.480 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2024-10-16 14:10:05.486 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.486 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.487 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.488 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2024-10-16 14:10:05.494 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.495 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.496 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.497 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2024-10-16 14:10:05.502 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.502 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.504 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.504 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2024-10-16 14:10:05.510 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.511 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.511 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.512 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2024-10-16 14:10:05.517 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.517 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.518 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.518 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2024-10-16 14:10:05.524 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.525 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.526 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.527 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2024-10-16 14:10:05.532 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.533 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.534 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.534 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2024-10-16 14:10:05.541 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.542 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.543 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.543 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2024-10-16 14:10:05.548 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.549 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.550 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.551 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2024-10-16 14:10:05.556 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.557 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.558 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.559 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2024-10-16 14:10:05.563 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.564 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.565 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.565 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2024-10-16 14:10:05.570 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.571 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.572 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.572 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2024-10-16 14:10:05.578 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.578 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.579 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.580 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2024-10-16 14:10:05.584 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.585 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.586 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.586 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2024-10-16 14:10:05.595 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.596 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.597 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.597 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2024-10-16 14:10:05.607 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.607 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.608 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.609 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2024-10-16 14:10:05.614 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.615 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.616 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.616 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2024-10-16 14:10:05.622 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.623 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.624 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.624 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2024-10-16 14:10:05.629 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.629 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.630 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.631 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2024-10-16 14:10:05.636 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.636 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.637 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.638 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2024-10-16 14:10:05.642 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.643 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.644 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.644 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2024-10-16 14:10:05.649 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.649 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.650 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.651 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2024-10-16 14:10:05.663 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.664 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.665 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.666 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2024-10-16 14:10:05.672 INFO 4486 --- [ocp-async-9,22bfab6a4959f195,ae52ca7a4e0e5314] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:05.700 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.702 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.703 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.704 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2024-10-16 14:10:05.713 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.714 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.715 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.716 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2024-10-16 14:10:05.724 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.725 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.726 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.727 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2024-10-16 14:10:05.737 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.738 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.739 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.739 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2024-10-16 14:10:05.743 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.744 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.744 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.745 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2024-10-16 14:10:05.749 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.750 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.751 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.752 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2024-10-16 14:10:05.756 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2024-10-16 14:10:05.756 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2024-10-16 14:10:05.764 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.765 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2024-10-16 14:10:05.769 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.769 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.771 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.772 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2024-10-16 14:10:05.780 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.781 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select now() 2024-10-16 14:10:05.782 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:05.782 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2024-10-16 14:10:05.786 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2024-10-16 14:10:05.795 INFO 4486 --- [subtask-executor32,791e68b70ade5cad,4daae26bdf5a3b61] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 7934617, operation:EXECUTE, state: SUCCESSFUL 2024-10-16 14:10:07.066 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:10:07.067 INFO 4486 --- [ocp-schedules-22,7aa2f412015d3e6d,0f7d5bb582b1b32b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:10:07.069 INFO 4486 --- [ocp-schedules-22,7aa2f412015d3e6d,0f7d5bb582b1b32b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:07.081 INFO 4486 --- [ocp-schedules-3,c1f4e7a151c4f1d7,2d08ef01ec6cc669] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:10:07.083 INFO 4486 --- [ocp-schedules-3,c1f4e7a151c4f1d7,2d08ef01ec6cc669] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:07.083 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:10:07.089 INFO 4486 --- [ocp-schedules-21,c08d14530b6444bf,71390097c8ba9062] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:10:07.089 INFO 4486 --- [ocp-schedules-4,4596ce17338cc3ad,f0daa8d6c035dd3c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:10:07.092 INFO 4486 --- [ocp-schedules-30,b33e397e5de4f5c8,4e894bff0b36fc5f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:10:07.092 INFO 4486 --- [host-check-clockdiff3,c08d14530b6444bf,10c940d2205882f7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:10:07.092 INFO 4486 --- [host-check-clockdiff1,c08d14530b6444bf,2a5c3585d2999ee1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:10:07.092 INFO 4486 --- [host-check-clockdiff4,c08d14530b6444bf,c2d2390c1fe5574e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:10:07.093 INFO 4486 --- [host-check-clockdiff2,c08d14530b6444bf,f202ec2da7385a2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:10:07.093 INFO 4486 --- [host-check-agent-status1,b33e397e5de4f5c8,cfd2df3c0eb26038] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:10:07.095 INFO 4486 --- [host-check-agent-status3,b33e397e5de4f5c8,0d2a24227e157886] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:10:07.095 INFO 4486 --- [host-check-agent-status2,b33e397e5de4f5c8,a81209195bafe9bd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:10:07.095 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:10:07.095 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:10:07.096 INFO 4486 --- [host-check-agent-status4,b33e397e5de4f5c8,6d9ad00e3b8d605a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:10:07.096 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:10:07.096 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:10:07.112 INFO 4486 --- [host-check-agent-status1,b33e397e5de4f5c8,cfd2df3c0eb26038] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:10:07.112 INFO 4486 --- [host-check-agent-status3,b33e397e5de4f5c8,0d2a24227e157886] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:10:07.108 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:10:07.113 INFO 4486 --- [host-check-agent-status2,b33e397e5de4f5c8,a81209195bafe9bd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:10:07.121 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:10:07.121 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:10:07.133 INFO 4486 --- [host-check-agent-status4,b33e397e5de4f5c8,6d9ad00e3b8d605a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:10:07.134 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:10:07.134 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:10:07.134 INFO 4486 --- [host-check-clockdiff1,c08d14530b6444bf,2a5c3585d2999ee1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059007 0 0 , error= 2024-10-16 14:10:07.134 INFO 4486 --- [host-check-clockdiff2,c08d14530b6444bf,f202ec2da7385a2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059007 4 4 , error= 2024-10-16 14:10:07.140 INFO 4486 --- [host-check-agent-status3,b33e397e5de4f5c8,0d2a24227e157886] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:10:07.140 INFO 4486 --- [host-check-agent-status2,b33e397e5de4f5c8,a81209195bafe9bd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:10:07.140 INFO 4486 --- [host-check-agent-status1,b33e397e5de4f5c8,cfd2df3c0eb26038] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:10:07.140 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:10:07.140 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:10:07.143 INFO 4486 --- [host-check-agent-status4,b33e397e5de4f5c8,6d9ad00e3b8d605a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:10:07.144 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:10:07.144 INFO 4486 --- [host-check-clockdiff3,c08d14530b6444bf,10c940d2205882f7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059007 4 4 , error= 2024-10-16 14:10:07.150 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.151 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:10:07.152 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:10:07.167 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.169 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:10:07.172 INFO 4486 --- [host-check-agent-status1,b33e397e5de4f5c8,cfd2df3c0eb26038] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:10:07.173 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:10:07.176 INFO 4486 --- [host-check-agent-status2,b33e397e5de4f5c8,a81209195bafe9bd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:10:07.176 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:10:07.176 INFO 4486 --- [host-check-agent-status4,b33e397e5de4f5c8,6d9ad00e3b8d605a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:10:07.177 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:10:07.177 INFO 4486 --- [host-check-agent-status3,b33e397e5de4f5c8,0d2a24227e157886] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:10:07.182 INFO 4486 --- [host-check-agent-status1,b33e397e5de4f5c8,cfd2df3c0eb26038] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:10:07.182 INFO 4486 --- [host-check-agent-status1,b33e397e5de4f5c8,cfd2df3c0eb26038] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:10:07.182 INFO 4486 --- [host-check-agent-status1,b33e397e5de4f5c8,cfd2df3c0eb26038] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:10:07.183 INFO 4486 --- [host-check-agent-status2,b33e397e5de4f5c8,a81209195bafe9bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:10:07.183 INFO 4486 --- [host-check-agent-status2,b33e397e5de4f5c8,a81209195bafe9bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:10:07.183 INFO 4486 --- [host-check-agent-status2,b33e397e5de4f5c8,a81209195bafe9bd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:10:07.184 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:10:07.187529898+08:00 2024-10-16 14:10:07.184 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:10:07.188060953+08:00 2024-10-16 14:10:07.185 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:10:07.184+08:00 2024-10-16 14:10:07.185 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:10:07.184+08:00 2024-10-16 14:10:07.186 INFO 4486 --- [host-check-agent-status3,b33e397e5de4f5c8,0d2a24227e157886] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:10:07.186 INFO 4486 --- [host-check-agent-status3,b33e397e5de4f5c8,0d2a24227e157886] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:10:07.186 INFO 4486 --- [host-check-agent-status3,b33e397e5de4f5c8,0d2a24227e157886] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:10:07.187 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:10:07.187 INFO 4486 --- [host-check-agent-status4,b33e397e5de4f5c8,6d9ad00e3b8d605a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:10:07.188 INFO 4486 --- [host-check-agent-status4,b33e397e5de4f5c8,6d9ad00e3b8d605a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:10:07.188 INFO 4486 --- [host-check-agent-status4,b33e397e5de4f5c8,6d9ad00e3b8d605a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:10:07.188 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:10:07.184+08:00 and status AVAILABLE 2024-10-16 14:10:07.188 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:10:07.187876473+08:00 2024-10-16 14:10:07.188 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:10:07.184+08:00 and status AVAILABLE 2024-10-16 14:10:07.189 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:10:07.188+08:00 2024-10-16 14:10:07.189 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:10:07.190 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:10:07.194156119+08:00 2024-10-16 14:10:07.190 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:10:07.191 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:10:07.190+08:00 2024-10-16 14:10:07.191 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:10:07.193 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:10:07.188+08:00 and status AVAILABLE 2024-10-16 14:10:07.194 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:10:07.190+08:00 and status AVAILABLE 2024-10-16 14:10:07.195 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:10:07.196 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:10:07.198 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.201 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:10:07.201 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:10:07.203 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:10:07.203 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:10:07.204 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:10:07.207 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:10:07.207 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:10:07.209 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:10:07.209 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:10:07.212 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.213 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:07.227 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.228 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:07.231 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:07.232 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:07.233 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:07.234 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:07.242 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.243 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:07.245 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.247 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:10:07.257 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:07.258 INFO 4486 --- [ocp-schedules-29,c5ffb478178e1e11,ab26e55c916ac7f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:10:07.259 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:10:07.262 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:10:07.275 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:10:07.276 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:10:07.276 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:10:07.289 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:10:07.291 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:10:07.295 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:07.296 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:07.300 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:10:07.308 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:07.317 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:07.324 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:10:07.327 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:10:07.336 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:10:07.337 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:10:07.345 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:10:07.351 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:07.352 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:07.355 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:10:07.372 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:10:07.374 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:07.384 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:10:07.385 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:10:07.389 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:10:07.396 INFO 4486 --- [host-check-host-status4,4596ce17338cc3ad,81d0b3b61bc89d2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:10:07.400 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:07.400 INFO 4486 --- [host-check-host-status3,4596ce17338cc3ad,ab3c5a88da5f6ea4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:10:07.412 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:10:07.424 INFO 4486 --- [host-check-host-status2,4596ce17338cc3ad,9c5c33073078db94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:10:07.453 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:10:07.465 INFO 4486 --- [host-check-host-status1,4596ce17338cc3ad,119072061d40bb9d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:10:07.672 INFO 4486 --- [ocp-async-5,6b1a4681c9a36f86,17a0e970ab8be94c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:07.695 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:07.994 INFO 4486 --- [prometheus-minute-schedule-1,c41d614ff60d3372,f5106cd8249202df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:08.000 INFO 4486 --- [prometheus-minute-schedule-1,c41d614ff60d3372,f5106cd8249202df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:08.056 INFO 4486 --- [host-check-clockdiff4,c08d14530b6444bf,c2d2390c1fe5574e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059008 4 4 , error= 2024-10-16 14:10:08.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8cf3d37729c52101,0d3c951904b86b6a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=8cf3d37729c52101, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:08.714 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,8cf3d37729c52101,0d3c951904b86b6a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=8cf3d37729c52101, duration=1 ms] 2024-10-16 14:10:09.158 INFO 4486 --- [ocp-schedules-24,6a9078af712ac5da,c2dfc4e0a41c7153] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2024-10-16 14:10:09.159 INFO 4486 --- [ocp-schedules-24,6a9078af712ac5da,c2dfc4e0a41c7153] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:09.672 INFO 4486 --- [ocp-async-6,4edd4ad35863df95,1f54b3fcfdb40307] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:09.893 INFO 4486 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:5266501, status:SUCCESS 2024-10-16 14:10:09.910 INFO 4486 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:5266501, status:SUCCESS 2024-10-16 14:10:10.006 INFO 4486 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:10:10.006 INFO 4486 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:10:10.028 INFO 4486 --- [Check obproxy2,877c0c699ca3f2e5,12ff302b42034bd0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:10:10.028 INFO 4486 --- [Check obproxy1,960a8309cec8ccc0,2d41f890678fd837] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:10:10.029 INFO 4486 --- [Check obproxy1,960a8309cec8ccc0,2d41f890678fd837] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:10.029 INFO 4486 --- [Check obproxy2,877c0c699ca3f2e5,12ff302b42034bd0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:10.035 INFO 4486 --- [Collect obproxy1,b901cd024807287f,986e7bbe50e2d6a1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:10:10.036 INFO 4486 --- [Collect obproxy1,b901cd024807287f,986e7bbe50e2d6a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:10.036 INFO 4486 --- [Collect obproxy2,aa5b7dbc0e38de2f,85aba4b32fbf33e7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:10:10.036 INFO 4486 --- [Collect obproxy2,aa5b7dbc0e38de2f,85aba4b32fbf33e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:10.109 INFO 4486 --- [Check obproxy2,877c0c699ca3f2e5,12ff302b42034bd0] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:10:10.109 INFO 4486 --- [Check obproxy2,877c0c699ca3f2e5,12ff302b42034bd0] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:10:10.112 INFO 4486 --- [Collect obproxy2,aa5b7dbc0e38de2f,85aba4b32fbf33e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:10:10.113 INFO 4486 --- [Collect obproxy1,b901cd024807287f,986e7bbe50e2d6a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:10:10.114 INFO 4486 --- [Check obproxy1,960a8309cec8ccc0,2d41f890678fd837] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:10:10.114 INFO 4486 --- [Check obproxy1,960a8309cec8ccc0,2d41f890678fd837] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:10:10.117 INFO 4486 --- [Collect obproxy1,b901cd024807287f,986e7bbe50e2d6a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:10:10.117 INFO 4486 --- [Collect obproxy2,aa5b7dbc0e38de2f,85aba4b32fbf33e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:10:10.120 INFO 4486 --- [Collect obproxy1,b901cd024807287f,986e7bbe50e2d6a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:10:10.122 INFO 4486 --- [Collect obproxy2,aa5b7dbc0e38de2f,85aba4b32fbf33e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:10:10.696 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:10.776 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c683853594eb1f09,b711051026ecba28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=c683853594eb1f09, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:10.777 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,c683853594eb1f09,b711051026ecba28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=c683853594eb1f09, duration=2 ms] 2024-10-16 14:10:10.976 INFO 4486 --- [async-queue-offer-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 60, queue size = 4, ruleSize = 4. 2024-10-16 14:10:10.994 INFO 4486 --- [prometheus-minute-schedule-1,7516c2482b669cfa,fe7f2cf9fe05319d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:11.000 INFO 4486 --- [prometheus-minute-schedule-1,7516c2482b669cfa,fe7f2cf9fe05319d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:11.006 INFO 4486 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 13, ruleSize = 13. 2024-10-16 14:10:11.526 INFO 4486 --- [ocp-schedules-18,d558afcb71e9e09a,3cd600aeb93ba780] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:10:11.594 INFO 4486 --- [ocp-schedules-23,e63509a6ec651ab5,f77c378593f2b8c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:10:11.605 WARN 4486 --- [scanner-13341591-1,d558afcb71e9e09a,49ef00c51a936bcd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:10:11.673 INFO 4486 --- [ocp-async-8,26c54b8f605a8e24,ca10dcd66550b9f8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:11.747 INFO 4486 --- [alarm-detect-12,d558afcb71e9e09a,fe6ce1bb618e8bb6] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729058985 ~ 1729059001 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729059000, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729059000, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729059000=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:10:11.931 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=10, affectRows=10 2024-10-16 14:10:11.931 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=21, affectRows=21 2024-10-16 14:10:11.932 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=69, affectRows=69 2024-10-16 14:10:11.932 INFO 4486 --- [metric-archive-second-26,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=10, validCount=10 2024-10-16 14:10:11.932 INFO 4486 --- [metric-archive-second-29,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=21, validCount=21 2024-10-16 14:10:11.933 INFO 4486 --- [metric-archive-second-18,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=68, validCount=68 2024-10-16 14:10:11.936 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=127, affectRows=127 2024-10-16 14:10:11.937 INFO 4486 --- [metric-archive-second-31,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=127, validCount=127 2024-10-16 14:10:11.939 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=138, affectRows=138 2024-10-16 14:10:11.940 INFO 4486 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=138, validCount=138 2024-10-16 14:10:11.941 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=133, affectRows=133 2024-10-16 14:10:11.941 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=105, affectRows=105 2024-10-16 14:10:11.941 INFO 4486 --- [metric-archive-second-25,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=133, validCount=133 2024-10-16 14:10:11.942 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=161, affectRows=161 2024-10-16 14:10:11.942 INFO 4486 --- [metric-archive-second-32,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=105, validCount=105 2024-10-16 14:10:11.943 INFO 4486 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=160, validCount=160 2024-10-16 14:10:11.945 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=181, affectRows=181 2024-10-16 14:10:11.946 INFO 4486 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=181, validCount=181 2024-10-16 14:10:11.946 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=232, affectRows=232 2024-10-16 14:10:11.946 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=128, affectRows=128 2024-10-16 14:10:11.947 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=209, affectRows=209 2024-10-16 14:10:11.949 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=182, affectRows=182 2024-10-16 14:10:11.949 INFO 4486 --- [metric-archive-second-19,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=128, validCount=128 2024-10-16 14:10:11.950 INFO 4486 --- [metric-archive-second-10,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=232, validCount=232 2024-10-16 14:10:11.950 INFO 4486 --- [metric-archive-second-20,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=182, validCount=182 2024-10-16 14:10:11.951 INFO 4486 --- [metric-archive-second-11,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=209, validCount=209 2024-10-16 14:10:11.953 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=259, affectRows=259 2024-10-16 14:10:11.954 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=174, affectRows=174 2024-10-16 14:10:11.955 INFO 4486 --- [metric-archive-second-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=174, validCount=174 2024-10-16 14:10:11.955 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=291, affectRows=291 2024-10-16 14:10:11.955 INFO 4486 --- [metric-archive-second-21,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=259, validCount=259 2024-10-16 14:10:11.955 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=323, affectRows=323 2024-10-16 14:10:11.957 INFO 4486 --- [metric-archive-second-27,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=291, validCount=291 2024-10-16 14:10:11.957 INFO 4486 --- [metric-archive-second-24,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=323, validCount=323 2024-10-16 14:10:11.962 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=309, affectRows=309 2024-10-16 14:10:11.964 INFO 4486 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=308, validCount=308 2024-10-16 14:10:11.971 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1729058940, nodeCount=398, affectRows=398 2024-10-16 14:10:11.973 INFO 4486 --- [metric-archive-second-23,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1729058940, nodeCount=398, validCount=398 2024-10-16 14:10:12.268 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6be29b0b16fb7269,292024ca0e2cd8e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6be29b0b16fb7269, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:12.268 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,6be29b0b16fb7269,292024ca0e2cd8e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6be29b0b16fb7269, duration=2 ms] 2024-10-16 14:10:12.555 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,de9852cb42ce7f71,3eca64cc3fe0bdaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=de9852cb42ce7f71, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:12.555 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,de9852cb42ce7f71,3eca64cc3fe0bdaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=de9852cb42ce7f71, duration=1 ms] 2024-10-16 14:10:13.040 INFO 4486 --- [ocp-async-5,0d0269072c76e639,a8ebb6d3b15a39a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:10:13.673 INFO 4486 --- [ocp-async-4,3ae0eabad7426857,89477b22e909bcfe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:13.697 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:13.994 INFO 4486 --- [prometheus-minute-schedule-1,eaf0b332e5ce583f,7fdbabb4f898a49e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:14.000 INFO 4486 --- [prometheus-minute-schedule-1,eaf0b332e5ce583f,7fdbabb4f898a49e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:14.024 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,8c2fd7d719bf1d60,f28463a2bbf432b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.178, traceId=8c2fd7d719bf1d60, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:10:14.025 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,8c2fd7d719bf1d60,f28463a2bbf432b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.178, traceId=8c2fd7d719bf1d60, duration=2 ms] 2024-10-16 14:10:15.578 INFO 4486 --- [metric-archive-async-28,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1729058940, nodeCount=477, affectRows=477 2024-10-16 14:10:15.673 INFO 4486 --- [ocp-async-1,fd380a39a331bb50,fe46ba1773e989e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:16.273 INFO 4486 --- [metric-archive-async-28,,] c.o.o.m.service.OcpMetricArchiveService : Rollup async second to minute, timestamp=1729058940, nodeCount=475, validCount=475 2024-10-16 14:10:16.698 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:16.994 INFO 4486 --- [prometheus-minute-schedule-1,8f9064e1adc745f8,98f56fff03108317] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:17.000 INFO 4486 --- [prometheus-minute-schedule-1,8f9064e1adc745f8,98f56fff03108317] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:17.070 INFO 4486 --- [ocp-schedules-22,7aa2f412015d3e6d,0f7d5bb582b1b32b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:10:17.173 INFO 4486 --- [ocp-schedules-3,c1f4e7a151c4f1d7,2d08ef01ec6cc669] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:10:17.336 INFO 4486 --- [ocp-schedules-8,9f82596ece2e0b52,353dd92d5e2e9a57] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:10:17.673 INFO 4486 --- [ocp-async-1,d7f38cd8d5676f55,f0fe7b9b92ce832d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:17.961 INFO 4486 --- [ocp-schedules-30,4d013f6c91b6786e,9a27876592368627] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2024-10-16 14:10:17.961 INFO 4486 --- [ocp-schedules-30,4d013f6c91b6786e,9a27876592368627] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2024-10-16 14:10:18.000 INFO 4486 --- [pool-285-thread-3,4d013f6c91b6786e,bd9a29ad1e041356] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:10:18.000 INFO 4486 --- [pool-285-thread-2,4d013f6c91b6786e,c0f443989ef760cb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:10:18.000 INFO 4486 --- [pool-285-thread-1,4d013f6c91b6786e,4656b3844a10a672] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2024-10-16 14:10:18.029 INFO 4486 --- [pool-285-thread-1,4d013f6c91b6786e,4656b3844a10a672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:18.030 INFO 4486 --- [pool-285-thread-2,4d013f6c91b6786e,c0f443989ef760cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:18.033 INFO 4486 --- [pool-285-thread-1,4d013f6c91b6786e,4656b3844a10a672] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729059017000000] 2024-10-16 14:10:18.036 INFO 4486 --- [pool-285-thread-2,4d013f6c91b6786e,c0f443989ef760cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729059017000000] 2024-10-16 14:10:18.039 INFO 4486 --- [pool-285-thread-3,4d013f6c91b6786e,bd9a29ad1e041356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:18.043 INFO 4486 --- [pool-285-thread-3,4d013f6c91b6786e,bd9a29ad1e041356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1729052097000000, 1729059017000000] 2024-10-16 14:10:18.322 WARN 4486 --- [pool-285-thread-3,4d013f6c91b6786e,bd9a29ad1e041356] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1001 2024-10-16 14:10:18.322 WARN 4486 --- [pool-285-thread-3,4d013f6c91b6786e,bd9a29ad1e041356] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:glob_3zone, ob tenant id:1003 2024-10-16 14:10:18.324 INFO 4486 --- [pool-285-thread-3,4d013f6c91b6786e,bd9a29ad1e041356] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:10:18.618 WARN 4486 --- [pool-285-thread-1,4d013f6c91b6786e,4656b3844a10a672] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1003 2024-10-16 14:10:18.618 WARN 4486 --- [pool-285-thread-1,4d013f6c91b6786e,4656b3844a10a672] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:OceanBase, ob tenant id:1009 2024-10-16 14:10:18.629 INFO 4486 --- [pool-285-thread-1,4d013f6c91b6786e,4656b3844a10a672] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:10:18.719 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2ba8d67ff6b29809,f7df5ede9d0d6cd9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=2ba8d67ff6b29809, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:18.719 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,2ba8d67ff6b29809,f7df5ede9d0d6cd9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=2ba8d67ff6b29809, duration=2 ms] 2024-10-16 14:10:18.900 WARN 4486 --- [pool-285-thread-2,4d013f6c91b6786e,c0f443989ef760cb] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:obce_3zones, ob tenant id:1009 2024-10-16 14:10:18.901 INFO 4486 --- [pool-285-thread-2,4d013f6c91b6786e,c0f443989ef760cb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2024-10-16 14:10:18.903 INFO 4486 --- [ocp-schedules-30,4d013f6c91b6786e,9a27876592368627] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2024-10-16 14:10:19.195 INFO 4486 --- [ocp-schedules-24,6a9078af712ac5da,c2dfc4e0a41c7153] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=3, successCount=3 2024-10-16 14:10:19.674 INFO 4486 --- [ocp-async-1,52fb3b3e40f3cde2,a77902b4141f82f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:19.689 INFO 4486 --- [ocp-schedules-15,6fa4a53d5d06e9b8,53dbf6b7c7dc5b24] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:10:19.699 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:19.994 INFO 4486 --- [prometheus-minute-schedule-1,679d07414bfe6c71,6b0cd3e852462f56] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:20.000 INFO 4486 --- [prometheus-minute-schedule-1,679d07414bfe6c71,6b0cd3e852462f56] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:20.781 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,29c037cbb809a605,bf84caaf45ec2b9f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=29c037cbb809a605, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:20.782 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,29c037cbb809a605,bf84caaf45ec2b9f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=29c037cbb809a605, duration=2 ms] 2024-10-16 14:10:20.814 INFO 4486 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Not the leader for service:PLAN_CHANGE_DETECTOR, skipped 2024-10-16 14:10:20.996 INFO 4486 --- [prometheus-cache-monitor1,71a3582b2793caf5,497a47f1ef2a4e77] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":13854,"minuteWriteQueue":0,"minuteReadCacheSize":8468,"secondWriteCacheSize":3450,"seriesIdKeySize":40024,"secondWriteQueue":0,"minuteWriteCacheSize":8187} 2024-10-16 14:10:20.996 INFO 4486 --- [prometheus-cache-monitor1,71a3582b2793caf5,497a47f1ef2a4e77] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{ob_sysstat_40012=41, ob_sysstat_40011=41, ob_server_resource_memory_assigned_bytes=23, ob_sysstat_40010=41, ob_unit_num=23, ob_database_disk_required_size=128, go_memstats_mspan_sys_bytes=28, go_memstats_stack_sys_bytes=28, ob_tenant_request_queue_total_size=44, jvm_gc_memory_allocated_bytes_total=4, ob_system_event_total_waits=54, ob_tenant_disk_v4_log_disk_size=32, ob_sysstat_40019=41, ob_sysstat_40018=41, processor_process_metrics_total=34, jvm_gc_max_data_size_bytes=4, go_memstats_frees_total=28, sql_audit_input_collect_data_jumped_total=11, host_available=28, mysql_output_write_sql_failed_count=31, ob_database_disk_data_size=127, ob_sysstat_190701=9, ob_sysstat_190702=9, ob_tenant_memstore_freeze_cnt=36, ob_tenant_unit_min_memory=54, ob_sysstat_60087=41, ob_memstore_total_bytes=44, metric_cardinality_count=41, jvm_threads_states_threads=9, process_resident_memory_bytes=28, ob_cache_size_bytes=54, ob_sysstat_190707=9, ob_tenant_unit_max_memory=54, input_collect_metrics_total=40, ob_sysstat_50039=39, ob_sysstat_60083=39, ob_sysstat_50038=41, node_disk_io_time_weighted_seconds_total=50, process_open_fds=28, ob_sysstat_50037=41, process_memory_rss_bytes=43, ob_server_resource_cpu_assigned_percent=23, ob_sysstat_50030=39, sql_audit_input_collect_lost_total=32, go_memstats_heap_inuse_bytes=28, ob_sysstat_40118=41, ob_sysstat_40117=41, ob_sysstat_40116=41, node_memory_memfree_bytes=34, ob_sysstat_40005=41, process_exists=53, ob_sysstat_40004=41, ob_sysstat_40003=41, ob_sysstat_40002=41, ob_sysstat_40001=41, go_gc_duration_seconds=33, ob_server_resource_disk_bytes=23, ob_sysstat_40000=41, odp_current_session=40, ob_sysstat_190601=9, ob_sysstat_190602=9, ob_sysstat_190603=9, process_thread_count=43, ob_sysstat_50040=39, ob_sysstat_40009=41, ob_sysstat_40008=41, ob_disk_free_bytes=23, ob_sysstat_40007=41, node_filesystem_avail_bytes=102, ob_sysstat_40006=41, ob_backup_storage_capacity_bytes=777, node_time_seconds=8, ob_unit_config_max_iops=82, ob_plan_cache_memory_bytes=38, node_network_transmit_err 2024-10-16 14:10:21.604 INFO 4486 --- [ocp-schedules-10,9904a6266bfa2536,b5d9724df289363d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:10:21.674 INFO 4486 --- [ocp-async-1,2aab58e4b5a14f53,a95b4f367cbaa9eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:21.793 INFO 4486 --- [ocp-schedules-16,e1537adbd98f2557,d820f89b4958229d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:10:21.794 INFO 4486 --- [ocp-schedules-16,85193bc6e9074f65,048a9a82da82abae] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2024-10-16 14:10:21.795 INFO 4486 --- [ocp-schedules-16,85193bc6e9074f65,048a9a82da82abae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:21.828 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:21.829 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:10:21.840 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:21.841 INFO 4486 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:10:21.866 INFO 4486 --- [ocp-schedules-21,490136c0a8c7f03f,dffcae23bf486050] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:10:21.867 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:21.868 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:21.869 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:10:21.869 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:21.871 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:10:21.893 WARN 4486 --- [scanner-13342421-1,490136c0a8c7f03f,9b2eeec31b6ae3de] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:10:21.925 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:21.926 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:21.927 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:10:21.928 WARN 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] idcName is empty, zoneName=zone1 2024-10-16 14:10:21.929 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:21.933 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:10:21.939 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:10:21.987 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:21.991 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:06+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:10:22.052 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.197.49.184, port=2882, sqlPort=2881, installPath=/root/oceanbase, runPath=/root/oceanbase, runUser=root, version=4.2.1.1_101010012023111012-2f6924cd5a576f09d6e7f212fac83f1a15ff531a(Nov 10 2023 12:13:59), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2023-11-30T11:17:05.768352+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:22.075 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:22.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:22.076 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:10:22.077 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster OceanBase 2024-10-16 14:10:22.082 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2024-10-16 14:10:22.088 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:10:22.090 INFO 4486 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:10:22.107 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:22.109 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:22.110 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2024-10-16 14:10:22.115 INFO 4486 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:10:22.128 INFO 4486 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] cluster is in operation, skip sync cluster, clusterId=5, status=UPGRADING 2024-10-16 14:10:22.128 INFO 4486 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=false, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:10:22.162 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:22.163 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:22.164 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2024-10-16 14:10:22.168 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:22.171 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2024-10-16 14:10:22.175 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:10:22.192 INFO 4486 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=20506, runPath=/root/oceanbase, runUser=root, execute=/root/oceanbase/bin/observer, installPath=/root/oceanbase) 2024-10-16 14:10:22.201 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:22.201 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:22.201 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000009, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:22.202 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:10:22.202 INFO 4486 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:10:22.211 INFO 4486 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:10:22.211 INFO 4486 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=OceanBase, obClusterId=1701313637, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"OceanBase","ObClusterId":1701313637,"ObRegion":"OceanBase","ObRegionId":1701313637,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.184:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058886919000}, createTime=2023-11-30T11:25:01+08:00, creator=admin, obVersion=4.2.1.1, packageOperatingSystem=null, loadType=null, tenantCount=4) 2024-10-16 14:10:22.228 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000012, ip=172.28.46.153, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000016, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:10:29.767051+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:22.229 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:10:22.253 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000008, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:22.253 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:44+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:10:22.272 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,71ac7eb04d99bb46,d5a859e3dccb2b18] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=71ac7eb04d99bb46, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:22.272 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,71ac7eb04d99bb46,d5a859e3dccb2b18] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=71ac7eb04d99bb46, duration=1 ms] 2024-10-16 14:10:22.281 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000011, ip=172.28.46.152, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=true, hostId=1000015, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:02:43.977764+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:22.281 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2024-10-16 14:10:22.310 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1000007, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:22.310 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:21+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00} 2024-10-16 14:10:22.338 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1000010, ip=172.28.46.151, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.2.1.9_109000032024092015-f2b0ead618247181e253e3912072c30ff965632d(Sep 20 2024 22:22:28), withRootserver=false, hostId=1000014, status=RUNNING, innerStatus=ACTIVE, startTime=2024-10-09T10:06:20.949477+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2024-10-16 14:10:22.365 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:22.366 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:22.367 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:10:22.371 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster glob_3zone 2024-10-16 14:10:22.376 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 2024-10-16 14:10:22.381 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000014 2024-10-16 14:10:22.383 INFO 4486 --- [Sync all clusters3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000014 2024-10-16 14:10:22.406 INFO 4486 --- [Sync all clusters3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.151:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2024-10-16 14:10:22.485 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,167f3c926c133bd8,dbe9233174676f4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.176, traceId=167f3c926c133bd8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:10:22.486 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,167f3c926c133bd8,dbe9233174676f4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.176, traceId=167f3c926c133bd8, duration=2 ms] 2024-10-16 14:10:22.546 INFO 4486 --- [Sync all clusters3,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1000014, sqlPort: 2881 got ObProcessInfoResult(pid=3818358, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2024-10-16 14:10:22.559 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,21306d5b510d4f80,ca8d7789018963f9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=21306d5b510d4f80, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:22.559 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,21306d5b510d4f80,ca8d7789018963f9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=21306d5b510d4f80, duration=1 ms] 2024-10-16 14:10:22.574 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:22.576 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:22.577 INFO 4486 --- [Sync all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2024-10-16 14:10:22.610 INFO 4486 --- [Sync all clusters3,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:10:22.610 INFO 4486 --- [Sync all clusters3,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1000003, name=glob_3zone, obClusterId=1701313644, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"glob_3zone","ObClusterId":1701313644,"ObRegion":"glob_3zone","ObRegionId":1701313644,"ReadonlyRsList":[],"RsList":[{"address":"172.28.46.152:2882","role":"LEADER","sql_port":2881},{"address":"172.28.46.151:2882","role":"FOLLOWER","sql_port":2881},{"address":"172.28.46.153:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058628393000}, createTime=2024-07-16T16:25:59+08:00, creator=admin, obVersion=4.2.1.9, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:10:22.700 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:22.994 INFO 4486 --- [prometheus-minute-schedule-1,5e6b8f84bf8ee75d,cf8a01f47a7a22c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:23.004 INFO 4486 --- [prometheus-minute-schedule-1,5e6b8f84bf8ee75d,cf8a01f47a7a22c8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:23.040 INFO 4486 --- [ocp-async-9,a007116f8afc248e,d55ee6de24a2fc69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:10:23.674 INFO 4486 --- [ocp-async-7,72c7d5a447a9e73d,19edc2cfe12fb37f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:24.294 INFO 4486 --- [ocp-schedules-30,4dca9121dea7fd81,d68ef6965de56d5a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 38 2024-10-16 14:10:25.508 INFO 4486 --- [ocp-schedules-11,64e68022e9fefbe1,66524d659a3ab322] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2024-10-16 14:10:25.674 INFO 4486 --- [ocp-async-2,8bcf347afe7f8e75,920008842254d59b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:25.701 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:25.994 INFO 4486 --- [prometheus-minute-schedule-1,04070c94e35bd3b1,ba4604e44db6ad6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:26.000 INFO 4486 --- [prometheus-minute-schedule-1,04070c94e35bd3b1,ba4604e44db6ad6d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:27.430 INFO 4486 --- [ocp-schedules-28,a28e09b7c60892cc,5c166afa452ea5aa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2024-10-16 14:10:27.430 INFO 4486 --- [ocp-schedules-28,a28e09b7c60892cc,5c166afa452ea5aa] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2024-10-16 14:10:27.462 INFO 4486 --- [Collect all tenants major compactions2,a28e09b7c60892cc,bf7db8701d38c3e0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:10:27.462 INFO 4486 --- [Collect all tenants major compactions1,a28e09b7c60892cc,1a682c83d4b7aa4d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:10:27.462 INFO 4486 --- [Collect all tenants major compactions3,a28e09b7c60892cc,12724d3709bf9f19] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:10:27.463 INFO 4486 --- [Collect all tenants major compactions4,a28e09b7c60892cc,8a779a0364efe524] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:10:27.463 INFO 4486 --- [Collect all tenants major compactions5,a28e09b7c60892cc,eb1bb9f88607a34c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:10:27.475 INFO 4486 --- [Collect all tenants major compactions1,a28e09b7c60892cc,1a682c83d4b7aa4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:27.476 INFO 4486 --- [Collect all tenants major compactions2,a28e09b7c60892cc,bf7db8701d38c3e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:27.477 INFO 4486 --- [Collect all tenants major compactions2,a28e09b7c60892cc,bf7db8701d38c3e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:10:27.477 INFO 4486 --- [Collect all tenants major compactions1,a28e09b7c60892cc,1a682c83d4b7aa4d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:10:27.481 INFO 4486 --- [Collect all tenants major compactions3,a28e09b7c60892cc,12724d3709bf9f19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:27.482 INFO 4486 --- [Collect all tenants major compactions2,a28e09b7c60892cc,bf7db8701d38c3e0] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015200761035432, broadcast scn:1729015200761035432, saved frozen scn:1729015200761035432 2024-10-16 14:10:27.482 INFO 4486 --- [Collect all tenants major compactions1,a28e09b7c60892cc,1a682c83d4b7aa4d] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015204717980573, broadcast scn:1729015204717980573, saved frozen scn:1729015204717980573 2024-10-16 14:10:27.482 INFO 4486 --- [Collect all tenants major compactions3,a28e09b7c60892cc,12724d3709bf9f19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1010] 2024-10-16 14:10:27.482 INFO 4486 --- [Collect all tenants major compactions1,a28e09b7c60892cc,1a682c83d4b7aa4d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:ocp_monitor 2024-10-16 14:10:27.482 INFO 4486 --- [Collect all tenants major compactions2,a28e09b7c60892cc,bf7db8701d38c3e0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:OceanBase, tenant:sys 2024-10-16 14:10:27.486 INFO 4486 --- [Collect all tenants major compactions3,a28e09b7c60892cc,12724d3709bf9f19] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:10:27.487 INFO 4486 --- [Collect all tenants major compactions3,a28e09b7c60892cc,12724d3709bf9f19] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:obce_3zones, tenant:yewu02 2024-10-16 14:10:27.496 INFO 4486 --- [Collect all tenants major compactions4,a28e09b7c60892cc,8a779a0364efe524] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:27.496 INFO 4486 --- [Collect all tenants major compactions5,a28e09b7c60892cc,eb1bb9f88607a34c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:27.498 INFO 4486 --- [Collect all tenants major compactions4,a28e09b7c60892cc,8a779a0364efe524] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2024-10-16 14:10:27.498 INFO 4486 --- [Collect all tenants major compactions5,a28e09b7c60892cc,eb1bb9f88607a34c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2024-10-16 14:10:27.502 INFO 4486 --- [Collect all tenants major compactions4,a28e09b7c60892cc,8a779a0364efe524] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202047756316, broadcast scn:1729015202047756316, saved frozen scn:1729015202047756316 2024-10-16 14:10:27.502 INFO 4486 --- [Collect all tenants major compactions4,a28e09b7c60892cc,8a779a0364efe524] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:yewu02 2024-10-16 14:10:27.503 INFO 4486 --- [Collect all tenants major compactions5,a28e09b7c60892cc,eb1bb9f88607a34c] o.i.t.TenantCompactionCollectServiceImpl : current compaction frozen scn:1729015202466959520, broadcast scn:1729015202466959520, saved frozen scn:1729015202466959520 2024-10-16 14:10:27.503 INFO 4486 --- [Collect all tenants major compactions5,a28e09b7c60892cc,eb1bb9f88607a34c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:glob_3zone, tenant:sys 2024-10-16 14:10:27.504 INFO 4486 --- [ocp-schedules-28,a28e09b7c60892cc,5c166afa452ea5aa] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2024-10-16 14:10:27.675 INFO 4486 --- [ocp-async-4,ff3b0dc3562f567d,bd51468aaf87135c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:28.187 INFO 4486 --- [ocp-schedules-25,9e987e2983e30f06,53e11fe011cbccf3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2024-10-16T14:10:28.187+08:00, clusters=[1, 5, 1000003] 2024-10-16 14:10:28.189 INFO 4486 --- [ocp-schedules-25,9e987e2983e30f06,53e11fe011cbccf3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster OceanBase, trans stat result is empty. 2024-10-16 14:10:28.189 INFO 4486 --- [ocp-schedules-25,9e987e2983e30f06,53e11fe011cbccf3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster obce_3zones, trans stat result is empty. 2024-10-16 14:10:28.190 INFO 4486 --- [ocp-schedules-25,9e987e2983e30f06,53e11fe011cbccf3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster glob_3zone, trans stat result is empty. 2024-10-16 14:10:28.702 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:28.726 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fc11225bd3fb7963,f72064d8b5a183d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fc11225bd3fb7963, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:28.726 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fc11225bd3fb7963,f72064d8b5a183d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fc11225bd3fb7963, duration=3 ms] 2024-10-16 14:10:28.994 INFO 4486 --- [prometheus-minute-schedule-1,46333a4d775a7e5b,043b87319dc33af6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:29.002 INFO 4486 --- [prometheus-minute-schedule-1,46333a4d775a7e5b,043b87319dc33af6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:29.675 INFO 4486 --- [ocp-async-8,559a9bd096bddf6c,39bc5a850e3aad83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:30.103 INFO 4486 --- [prometheus-second-schedule-1,f43a7b396096a01a,3b0e47bc123a48ba] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=23, collectTaskCount=23 2024-10-16 14:10:30.699 INFO 4486 --- [ocp-schedules-2,55759c81222c9267,892af8070423c324] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2024-10-16 14:10:30.701 INFO 4486 --- [ocp-schedules-2,55759c81222c9267,892af8070423c324] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:30.736 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:10:30.737 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.741 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:10:30.749 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.750 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:10:30.753 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:10:30.754 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.757 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.758 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:10:30.758 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:10:30.765 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.766 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:10:30.774 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.774 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:10:30.777 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.777 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:10:30.782 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.782 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:30.786 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ff197b94c2a020ef,6a2d230a41f4955f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=ff197b94c2a020ef, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:30.786 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,ff197b94c2a020ef,6a2d230a41f4955f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=ff197b94c2a020ef, duration=1 ms] 2024-10-16 14:10:30.788 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:10:30.788 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.788 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.790 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:10:30.792 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.793 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:30.793 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2024-10-16 14:10:30.801 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.803 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:30.805 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.805 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:10:30.811 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.812 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:10:30.814 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.814 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:10:30.817 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.818 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:30.823 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.824 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:10:30.831 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.832 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:10:30.832 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.833 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.834 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.835 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.835 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.835 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.836 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.836 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2024-10-16 14:10:30.837 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.838 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.838 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.839 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.840 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.840 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.841 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.841 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:10:30.841 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.842 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.842 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.843 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1010] 2024-10-16 14:10:30.843 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.844 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.844 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.845 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.846 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.846 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.847 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.848 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.848 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.849 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.850 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2024-10-16 14:10:30.850 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.851 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:10:30.853 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:10:30.860 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.861 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2024-10-16 14:10:30.866 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.867 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:30.867 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.868 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.870 INFO 4486 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=OceanBase, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:30.871 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.871 INFO 4486 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:10:30.873 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.874 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.876 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.877 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.877 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:30.878 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.880 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=48, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=48, userWhitelist=% 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:30.881 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2024-10-16 14:10:30.883 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.884 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.885 INFO 4486 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2024-10-16 14:10:30.885 INFO 4486 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=OceanBase, obClusterId=1701313637 2024-10-16 14:10:30.887 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.887 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.888 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.888 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:30.890 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.892 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.894 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.895 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.899 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.900 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1010, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.901 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=5 2024-10-16 14:10:30.904 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:10:30.919 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.921 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.924 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.925 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.928 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.929 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.932 INFO 4486 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:30.932 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.933 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.934 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.934 INFO 4486 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:10:30.937 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.939 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.942 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.943 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.946 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.947 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2024-10-16 14:10:30.950 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=50, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:30.950 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=50, userWhitelist=% 2024-10-16 14:10:30.950 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=52, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:30.950 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=52, userWhitelist=% 2024-10-16 14:10:30.950 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=67, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:30.950 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=67, userWhitelist=% 2024-10-16 14:10:30.950 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.951 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2024-10-16 14:10:30.954 INFO 4486 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=5 2024-10-16 14:10:30.954 INFO 4486 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=5, clusterName=obce_3zones, obClusterId=1701313639 2024-10-16 14:10:30.955 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:30.956 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2024-10-16 14:10:30.958 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1000003 2024-10-16 14:10:30.961 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.197.49.180,10.197.49.184 2024-10-16 14:10:31.016 INFO 4486 --- [Sync all tenant information3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=glob_3zone, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:31.017 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:31.018 INFO 4486 --- [Sync all tenant information3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2024-10-16 14:10:31.030 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000013, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:31.030 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000013, userWhitelist=% 2024-10-16 14:10:31.030 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000014, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:31.030 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000014, userWhitelist=% 2024-10-16 14:10:31.030 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1000015, realWhitelist=%, defaultWhitelist=127.0.0.1,10.197.49.180,10.197.49.184, userWhitelist=% 2024-10-16 14:10:31.030 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1000015, userWhitelist=% 2024-10-16 14:10:31.034 INFO 4486 --- [Sync all tenant information3,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1000003 2024-10-16 14:10:31.034 INFO 4486 --- [Sync all tenant information3,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1000003, clusterName=glob_3zone, obClusterId=1701313644 2024-10-16 14:10:31.611 INFO 4486 --- [ocp-schedules-6,d65b77b08913ac96,d7179b8d3b780d84] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:10:31.675 INFO 4486 --- [ocp-async-9,f082480e4c2c4e7f,b027c4bd49609b50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:31.703 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:31.994 INFO 4486 --- [prometheus-minute-schedule-1,108217aa4e90014b,9d0885a239a99813] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:32.000 INFO 4486 --- [prometheus-minute-schedule-1,108217aa4e90014b,9d0885a239a99813] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:32.076 INFO 4486 --- [ocp-schedules-14,d05a9cef01b92ff4,d7af9e09b12f81bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:10:32.120 WARN 4486 --- [scanner-13342657-1,d05a9cef01b92ff4,f2066edd8aea4f29] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:10:32.213 INFO 4486 --- [alarm-detect-18,d05a9cef01b92ff4,b84ddbebbe55b431] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729059000 ~ 1729059022 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729059015, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729059015, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729059015=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:10:32.276 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,61708d2cacce0d2d,f7b5dade6907c1e5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=61708d2cacce0d2d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:32.277 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,61708d2cacce0d2d,f7b5dade6907c1e5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=61708d2cacce0d2d, duration=1 ms] 2024-10-16 14:10:32.563 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,eb3b4e3883d70f8e,089444888c848e15] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=eb3b4e3883d70f8e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:32.563 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,eb3b4e3883d70f8e,089444888c848e15] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=eb3b4e3883d70f8e, duration=1 ms] 2024-10-16 14:10:33.041 INFO 4486 --- [ocp-async-6,c56220e1dc6d1ee4,d9a06a5e5c75a2fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:10:33.605 INFO 4486 --- [ocp-schedules-9,bcba2a2ac171a13f,dd9b051a5a290b98] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2024-10-16 14:10:33.607 INFO 4486 --- [ocp-schedules-9,bcba2a2ac171a13f,dd9b051a5a290b98] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:33.608 INFO 4486 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:10:33.608 INFO 4486 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:10:33.608 INFO 4486 --- [Collect compaction for all clusters3,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2024-10-16 14:10:33.675 INFO 4486 --- [ocp-async-10,8bc3eec2de833aaf,95c65c98c5cdac4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:34.704 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:34.994 INFO 4486 --- [prometheus-minute-schedule-1,b99b7bf0790b6f9e,e8acbe0289646e76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:35.002 INFO 4486 --- [prometheus-minute-schedule-1,b99b7bf0790b6f9e,e8acbe0289646e76] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:35.676 INFO 4486 --- [ocp-async-3,4431e1debf99cc13,c3d44a61ea56b883] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:37.063 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2024-10-16 14:10:37.064 INFO 4486 --- [ocp-schedules-26,59a2e27fdbb45fa8,d3ce4641000ba7fe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2024-10-16 14:10:37.066 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2024-10-16 14:10:37.066 INFO 4486 --- [ocp-schedules-6,b7b304d9bb787063,c8ddbcf6c105e4dc] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2024-10-16 14:10:37.068 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2024-10-16 14:10:37.069 INFO 4486 --- [ocp-schedules-6,b7b304d9bb787063,c8ddbcf6c105e4dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:37.070 INFO 4486 --- [ocp-schedules-26,59a2e27fdbb45fa8,d3ce4641000ba7fe] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2024-10-16 14:10:37.081 INFO 4486 --- [ocp-schedules-4,023ea23cf6f7799d,f207e44dfdb18783] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2024-10-16 14:10:37.083 INFO 4486 --- [ocp-schedules-4,023ea23cf6f7799d,f207e44dfdb18783] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 5, 1000003] 2024-10-16 14:10:37.084 INFO 4486 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : skip cluster not running and unavailable, cluster=ObClusterOverview(id=5, name=obce_3zones, obClusterId=1701313639, vpcId=1, vpcName=system-default, type=PRIMARY, status=UPGRADING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"obce_3zones","ObClusterId":1701313639,"ObRegion":"obce_3zones","ObRegionId":1701313639,"ReadonlyRsList":[],"RsList":[{"address":"10.197.49.178:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.197.49.176:2882","role":"LEADER","sql_port":2881},{"address":"10.197.49.177:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1729058940310000}, createTime=2023-12-01T13:33:59+08:00, creator=admin, obVersion=4.2.1.5, packageOperatingSystem=null, loadType=null, tenantCount=3) 2024-10-16 14:10:37.090 INFO 4486 --- [ocp-schedules-8,00f25437317ee70b,b127d57444bd90df] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:10:37.091 INFO 4486 --- [ocp-schedules-29,35596762020b7341,2032aaade6c23d26] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:10:37.091 INFO 4486 --- [ocp-schedules-21,f07578ce60131e21,5140c06367bbd04e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.197.49.184, 10.197.49.178, 10.197.49.180, 172.28.46.152] 2024-10-16 14:10:37.091 INFO 4486 --- [host-check-agent-status1,00f25437317ee70b,5ef01591fa7c5e38] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:10:37.091 INFO 4486 --- [host-check-agent-status2,00f25437317ee70b,d2d16138a3848566] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:10:37.092 INFO 4486 --- [host-check-agent-status3,00f25437317ee70b,490575bade527c6e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:10:37.092 INFO 4486 --- [host-check-agent-status4,00f25437317ee70b,d950edc8a49a53a5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:10:37.093 INFO 4486 --- [host-check-clockdiff1,f07578ce60131e21,9cb5918bf809ff8b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.184], timeoutMillis=20000 2024-10-16 14:10:37.093 INFO 4486 --- [host-check-clockdiff2,f07578ce60131e21,5c01f9cd64829ee1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.178], timeoutMillis=20000 2024-10-16 14:10:37.093 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.197.49.184, originalStatus=ONLINE 2024-10-16 14:10:37.093 INFO 4486 --- [host-check-clockdiff3,f07578ce60131e21,00f00c2b45d869af] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.197.49.180], timeoutMillis=20000 2024-10-16 14:10:37.093 INFO 4486 --- [host-check-clockdiff4,f07578ce60131e21,fdb17399023cca8d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[172.28.46.152], timeoutMillis=20000 2024-10-16 14:10:37.095 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1000015, ip=172.28.46.152, originalStatus=ONLINE 2024-10-16 14:10:37.096 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.197.49.178, originalStatus=ONLINE 2024-10-16 14:10:37.096 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=7, ip=10.197.49.180, originalStatus=AVAILABLE 2024-10-16 14:10:37.101 INFO 4486 --- [host-check-agent-status3,00f25437317ee70b,490575bade527c6e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:10:37.101 INFO 4486 --- [host-check-agent-status2,00f25437317ee70b,d2d16138a3848566] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:10:37.102 INFO 4486 --- [host-check-agent-status1,00f25437317ee70b,5ef01591fa7c5e38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:10:37.102 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:10:37.102 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:10:37.103 INFO 4486 --- [host-check-agent-status4,00f25437317ee70b,d950edc8a49a53a5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:10:37.104 INFO 4486 --- [host-check-agent-status1,00f25437317ee70b,5ef01591fa7c5e38] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:10:37.104 INFO 4486 --- [host-check-agent-status3,00f25437317ee70b,490575bade527c6e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:10:37.104 INFO 4486 --- [host-check-agent-status2,00f25437317ee70b,d2d16138a3848566] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:10:37.104 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:10:37.105 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:10:37.143 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:10:37.143 INFO 4486 --- [host-check-agent-status4,00f25437317ee70b,d950edc8a49a53a5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:10:37.143 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:10:37.160 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:10:37.161 INFO 4486 --- [host-check-clockdiff2,f07578ce60131e21,5c01f9cd64829ee1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059037 4 4 , error= 2024-10-16 14:10:37.161 INFO 4486 --- [host-check-clockdiff1,f07578ce60131e21,9cb5918bf809ff8b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059037 0 0 , error= 2024-10-16 14:10:37.167 INFO 4486 --- [host-check-clockdiff4,f07578ce60131e21,fdb17399023cca8d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059037 4 4 , error= 2024-10-16 14:10:37.171 INFO 4486 --- [host-check-clockdiff3,f07578ce60131e21,00f00c2b45d869af] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1729059037 4 4 , error= 2024-10-16 14:10:37.172 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:10:37.172 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.173 INFO 4486 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:10:37.174 INFO 4486 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.2.1.1 2024-10-16 14:10:37.193 INFO 4486 --- [host-check-agent-status2,00f25437317ee70b,d2d16138a3848566] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/agent/status, params:null 2024-10-16 14:10:37.195 INFO 4486 --- [host-check-agent-status1,00f25437317ee70b,5ef01591fa7c5e38] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/agent/status, params:null 2024-10-16 14:10:37.196 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/time, params:null 2024-10-16 14:10:37.197 INFO 4486 --- [host-check-agent-status4,00f25437317ee70b,d950edc8a49a53a5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/agent/status, params:null 2024-10-16 14:10:37.198 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.198 INFO 4486 --- [host-check-agent-status3,00f25437317ee70b,490575bade527c6e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.180:62888/api/v1/agent/status, params:null 2024-10-16 14:10:37.199 INFO 4486 --- [Check status for all clusters3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: select max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2024-10-16 14:10:37.199 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=7, ip=10.197.49.180, timestamp=2024-10-16T14:10:37.202465925+08:00 2024-10-16 14:10:37.199 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://172.28.46.152:62888/api/v1/time, params:null 2024-10-16 14:10:37.200 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=7, lastAvailableTime=2024-10-16T14:10:37.199+08:00 2024-10-16 14:10:37.200 INFO 4486 --- [host-check-agent-status2,00f25437317ee70b,d2d16138a3848566] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.178 2024-10-16 14:10:37.200 INFO 4486 --- [host-check-agent-status2,00f25437317ee70b,d2d16138a3848566] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.178 2024-10-16 14:10:37.200 INFO 4486 --- [host-check-agent-status2,00f25437317ee70b,d2d16138a3848566] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.178 2024-10-16 14:10:37.200 INFO 4486 --- [host-check-agent-status1,00f25437317ee70b,5ef01591fa7c5e38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.184 2024-10-16 14:10:37.200 INFO 4486 --- [host-check-agent-status1,00f25437317ee70b,5ef01591fa7c5e38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.184 2024-10-16 14:10:37.200 INFO 4486 --- [host-check-agent-status1,00f25437317ee70b,5ef01591fa7c5e38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.184 2024-10-16 14:10:37.201 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1000015, ip=172.28.46.152, timestamp=2024-10-16T14:10:37.204855511+08:00 2024-10-16 14:10:37.202 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1000015, lastAvailableTime=2024-10-16T14:10:37.201+08:00 2024-10-16 14:10:37.202 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.184:62888/api/v1/time, params:null 2024-10-16 14:10:37.202 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 7 last available time 2024-10-16T14:10:37.199+08:00 and status AVAILABLE 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.197.49.184, timestamp=2024-10-16T14:10:37.202565346+08:00 2024-10-16 14:10:37.203 INFO 4486 --- [Check status for all clusters3,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1000003, obVersion=4.2.1.9 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-agent-status3,00f25437317ee70b,490575bade527c6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.197.49.180 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-agent-status3,00f25437317ee70b,490575bade527c6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.197.49.180 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-agent-status3,00f25437317ee70b,490575bade527c6e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.197.49.180 2024-10-16 14:10:37.203 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-agent-status4,00f25437317ee70b,d950edc8a49a53a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=172.28.46.152 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-agent-status4,00f25437317ee70b,d950edc8a49a53a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=172.28.46.152 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-agent-status4,00f25437317ee70b,d950edc8a49a53a5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=172.28.46.152 2024-10-16 14:10:37.203 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2024-10-16T14:10:37.203+08:00 2024-10-16 14:10:37.204 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1000007 last available time 2024-10-16T14:10:37.201+08:00 and status AVAILABLE 2024-10-16 14:10:37.204 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:10:37.205 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=7 2024-10-16 14:10:37.206 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1000015 2024-10-16 14:10:37.206 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.206 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2024-10-16T14:10:37.203+08:00 and status AVAILABLE 2024-10-16 14:10:37.207 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.207 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.197.49.178:62888/api/v1/time, params:null 2024-10-16 14:10:37.208 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:10:37.208 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.197.49.178, timestamp=2024-10-16T14:10:37.212328106+08:00 2024-10-16 14:10:37.208 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2024-10-16 14:10:37.209 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2024-10-16T14:10:37.208+08:00 2024-10-16 14:10:37.209 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.212 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2024-10-16T14:10:37.208+08:00 and status AVAILABLE 2024-10-16 14:10:37.214 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=7 2024-10-16 14:10:37.214 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2024-10-16 14:10:37.214 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1004] 2024-10-16 14:10:37.215 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1000015 2024-10-16 14:10:37.216 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 7 2024-10-16 14:10:37.216 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1000015 2024-10-16 14:10:37.216 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2024-10-16 14:10:37.217 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.217 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.217 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.217 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.217 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.218 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.218 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2024-10-16 14:10:37.222 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.223 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:37.224 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2024-10-16 14:10:37.226 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2024-10-16 14:10:37.232 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.233 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:37.234 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'FAILED' AND end_time > date_sub(now(), INTERVAL ? DAY), args: [2] 2024-10-16 14:10:37.240 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:37.240 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:37.246 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.247 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2024-10-16 14:10:37.248 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:37.250 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.251 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT end_scn FROM oceanbase.GV$OB_LOG_STAT WHERE tenant_id = ? AND ls_id = ? AND role = 'leader' 2024-10-16 14:10:37.254 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2024-10-16 14:10:37.261 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.262 INFO 4486 --- [ocp-schedules-13,cd7536d659eb91d4,ed60b009590815de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, ls_id, sync_lsn, sync_scn, sync_status, err_code, comment FROM V$OB_LS_LOG_RESTORE_STATUS WHERE tenant_id = ?, args: [1004] 2024-10-16 14:10:37.272 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=40 2024-10-16 14:10:37.280 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000019 2024-10-16 14:10:37.283 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=40 2024-10-16 14:10:37.292 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000019 2024-10-16 14:10:37.294 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=49 2024-10-16 14:10:37.303 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:37.304 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=55 2024-10-16 14:10:37.306 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=49 2024-10-16 14:10:37.310 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:37.315 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=55 2024-10-16 14:10:37.324 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:37.332 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=41 2024-10-16 14:10:37.334 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2024-10-16 14:10:37.344 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000020 2024-10-16 14:10:37.345 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=41 2024-10-16 14:10:37.355 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000020 2024-10-16 14:10:37.362 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.180:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:37.373 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://172.28.46.152:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:37.378 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=50 2024-10-16 14:10:37.388 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=56 2024-10-16 14:10:37.391 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=50 2024-10-16 14:10:37.392 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=42 2024-10-16 14:10:37.399 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=56 2024-10-16 14:10:37.404 INFO 4486 --- [host-check-host-status3,35596762020b7341,69a2ad39c3fcd5c6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=42 2024-10-16 14:10:37.409 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1000021 2024-10-16 14:10:37.409 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.178:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:37.416 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.197.49.184:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2024-10-16 14:10:37.421 INFO 4486 --- [host-check-host-status4,35596762020b7341,b13dcae12ae428e7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1000021 2024-10-16 14:10:37.455 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=51 2024-10-16 14:10:37.459 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:37.461 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.462 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:10:37.464 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.464 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.465 INFO 4486 --- [host-check-host-status2,35596762020b7341,10b1de809a005be9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=51 2024-10-16 14:10:37.465 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:10:37.476 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=57 2024-10-16 14:10:37.482 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.482 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.482 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.482 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.482 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.483 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.487 INFO 4486 --- [host-check-host-status1,35596762020b7341,172ab9acddb8d260] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=57 2024-10-16 14:10:37.509 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id RIGHT JOIN (SELECT max(job_id) as _job_id, tenant_id as _tenant_id FROM ( SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, tenant_id, backup_type, status FROM CDB_OB_BACKUP_JOB_HISTORY ) WHERE tenant_id != 1 AND backup_type = 'FULL' AND status = 'COMPLETED' GROUP BY tenant_id ) AS t ON job_id = t._job_id AND tenant_id = t._tenant_id ORDER BY start_time DESC 2024-10-16 14:10:37.543 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : maxAlarmDays=2, failedTasks=[] 2024-10-16 14:10:37.589 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:37.607 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:37.608 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.609 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'time_zone' 2024-10-16 14:10:37.610 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.operation.BackupObOpsService : systemTenantZoneId=+08:00 2024-10-16 14:10:37.611 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.611 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:10:37.612 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.613 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.613 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:10:37.623 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.623 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.623 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.623 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.623 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.623 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.631 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, IF(start_scn_display != '', start_scn_display, NULL) AS min_first_time, IF(checkpoint_scn_display != '', checkpoint_scn_display, NULL) AS max_next_time, status, IF(checkpoint_scn != '', truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4), NULL) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY RIGHT JOIN (SELECT tenant_id as _tenant_id, max(round_id) as _round_id FROM CDB_OB_ARCHIVELOG_SUMMARY GROUP BY _tenant_id ) AS t ON tenant_id = t._tenant_id and round_id = t._round_id) WHERE delay >= ?, args: [900] 2024-10-16 14:10:37.676 INFO 4486 --- [ocp-async-9,54a7b882d116ade1,316f9ef581977af5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:37.706 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:37.821 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : minDelaySeconds=900, delayTasks=[] 2024-10-16 14:10:37.868 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:37.870 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.870 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:10:37.871 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.872 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.873 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:10:37.883 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.883 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.883 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.883 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.883 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:37.884 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:37.902 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE (status = 'DOING' OR status = 'INIT') AND start_time < date_sub(now(), INTERVAL ? MINUTE), args: [60] 2024-10-16 14:10:37.963 INFO 4486 --- [print-time-series-interval60-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:10:37.992 INFO 4486 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2024-10-16 14:10:37.994 INFO 4486 --- [prometheus-minute-schedule-1,4f6fb7e9576a3b97,4ec07b897679aa0c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:38.005 INFO 4486 --- [prometheus-minute-schedule-1,4f6fb7e9576a3b97,4ec07b897679aa0c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:38.127 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : minTimeoutMinutes=60, timeoutTasks=[] 2024-10-16 14:10:38.170 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=obce_3zones, tenantName=sys, dbUser=ocp_monitor 2024-10-16 14:10:38.172 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:38.173 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'system_time_zone' 2024-10-16 14:10:38.174 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:38.176 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:38.178 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT tenant_id, name, value, gmt_create, gmt_modified FROM __all_virtual_sys_variable WHERE name = ? 2024-10-16 14:10:38.188 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:38.188 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:38.188 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:38.188 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:38.188 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.b.c.o.o.c.MysqlClusterOperator : get system time zone: +08:00 2024-10-16 14:10:38.189 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2024-10-16 14:10:38.256 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT * FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED' AND end_time < date_sub(now(), INTERVAL ? DAY) AND end_time IN ( SELECT max(end_time) FROM (SELECT job_id, incarnation, job.tenant_id AS tenant_id, job.backup_set_id AS backup_set_id, backup_type, path AS backup_dest, start_timestamp AS start_time, end_timestamp AS end_time, now(6) AS check_time, status, comment, description, 'TENANT' AS backup_level, file.min_restore_scn_display AS snapshot_version_time, output_bytes FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id, min_restore_scn_display, output_bytes FROM CDB_OB_BACKUP_SET_FILES) file ON job.tenant_id = file.tid AND job.backup_set_id = file.backup_set_id WHERE tenant_id != 1) WHERE status = 'COMPLETED'), args: [2] 2024-10-16 14:10:38.478 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : minAlarmDays=2, tooOldTasks=[DataBackupHistoryEntity(clusterName=obce_3zones, obClusterId=1701313639, tenantId=1010, currentTime=2024-10-16 14:10:38.260121, startTime=2024-10-08 04:00:05.766953, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED)] 2024-10-16 14:10:38.478 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : alarmDays=2, tooOldTask=DataBackupInnerWrapperTask(clusterName=obce_3zones, obClusterId=1701313639, obTenantId=1010, currentTime=2024-10-16 14:10:38.260121, finishTime=2024-10-08 04:04:28.124077, agentServerIp=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, errorMsg=, status=COMPLETED) 2024-10-16 14:10:38.518 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : send alarm, alarmType:base_backup_too_long_time_no_one_success_task, target:null, labels:class OcpLabels { {tenant_name=yewu02, last_success_time=2024-10-08 04:04:28.124077, no_success_days_threshold=2, ob_cluster_name=obce_3zones, svr_ip=10.197.49.176,10.197.49.178,10.197.49.177,10.197.49.171,10.197.49.172,10.197.49.173, ob_cluster_id=1701313639, current_time=2024-10-16 14:10:38.260121} } 2024-10-16 14:10:38.575 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : cluster=obce_3zones does not enable secondary backup 2024-10-16 14:10:38.584 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:10:38.587 INFO 4486 --- [ocp-schedules-30,5f8b0cc1140ca4ae,25a1586c7b93ab21] c.o.o.b.i.alarm.BackupAlarmService : search root dir:[file:///nfsback/backup02/data/], metricValues size:[0]. 2024-10-16 14:10:38.734 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,d16ab955cc187983,7a070cd0a5b14e5e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=d16ab955cc187983, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:38.734 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,d16ab955cc187983,7a070cd0a5b14e5e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=d16ab955cc187983, duration=3 ms] 2024-10-16 14:10:39.676 INFO 4486 --- [ocp-async-4,9e99f02f64d0c1fc,cadb844e37163b40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:39.693 INFO 4486 --- [ocp-schedules-27,9838baba2bd2c4c9,9e9eaf58b3ca15b9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2024-10-16 14:10:40.007 INFO 4486 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=2 2024-10-16 14:10:40.007 INFO 4486 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=2 2024-10-16 14:10:40.028 INFO 4486 --- [Check obproxy2,251a58de5e3d1f1f,13d2c10c64f1ef92] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:10:40.028 INFO 4486 --- [Check obproxy2,251a58de5e3d1f1f,13d2c10c64f1ef92] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:40.029 INFO 4486 --- [Check obproxy1,251f5bf7d8a4f1c5,7c8e8cd80428a374] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:10:40.029 INFO 4486 --- [Check obproxy1,251f5bf7d8a4f1c5,7c8e8cd80428a374] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:40.032 INFO 4486 --- [Collect obproxy2,15454baedcf1f6cb,e4d9b503c7a54aac] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 172.28.46.153 2883 2024-10-16 14:10:40.032 INFO 4486 --- [Collect obproxy2,15454baedcf1f6cb,e4d9b503c7a54aac] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:40.032 INFO 4486 --- [Collect obproxy1,c981758513078d8e,a962a89068f85ead] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.197.49.176 2883 2024-10-16 14:10:40.032 INFO 4486 --- [Collect obproxy1,c981758513078d8e,a962a89068f85ead] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2024-10-16 14:10:40.050 INFO 4486 --- [Check obproxy2,251a58de5e3d1f1f,13d2c10c64f1ef92] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1000005 success, update last active time to now 2024-10-16 14:10:40.050 INFO 4486 --- [Check obproxy2,251a58de5e3d1f1f,13d2c10c64f1ef92] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1000005 proxy_id: null, need to sync to meta: false 2024-10-16 14:10:40.052 INFO 4486 --- [Collect obproxy2,15454baedcf1f6cb,e4d9b503c7a54aac] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:10:40.053 INFO 4486 --- [Check obproxy1,251f5bf7d8a4f1c5,7c8e8cd80428a374] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 5 success, update last active time to now 2024-10-16 14:10:40.053 INFO 4486 --- [Check obproxy1,251f5bf7d8a4f1c5,7c8e8cd80428a374] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 5 proxy_id: null, need to sync to meta: false 2024-10-16 14:10:40.056 INFO 4486 --- [Collect obproxy2,15454baedcf1f6cb,e4d9b503c7a54aac] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:10:40.058 INFO 4486 --- [Collect obproxy1,c981758513078d8e,a962a89068f85ead] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2024-10-16 14:10:40.061 INFO 4486 --- [Collect obproxy2,15454baedcf1f6cb,e4d9b503c7a54aac] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:10:40.063 INFO 4486 --- [Collect obproxy1,c981758513078d8e,a962a89068f85ead] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2024-10-16 14:10:40.068 INFO 4486 --- [Collect obproxy1,c981758513078d8e,a962a89068f85ead] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2024-10-16 14:10:40.706 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:40.790 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5d0bfe68eb7e655c,dff500d93dd9c442] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=5d0bfe68eb7e655c, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:40.790 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,5d0bfe68eb7e655c,dff500d93dd9c442] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=5d0bfe68eb7e655c, duration=1 ms] 2024-10-16 14:10:40.994 INFO 4486 --- [prometheus-minute-schedule-1,6e8272196c8e006f,10ce498dfd50d67c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:40.995 INFO 4486 --- [prometheus-hour-schedule-1,b0ceb2856a1999f4,f0acf3baa2f909f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729062000, interval=3600 2024-10-16 14:10:41.000 INFO 4486 --- [prometheus-minute-schedule-1,6e8272196c8e006f,10ce498dfd50d67c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:41.616 INFO 4486 --- [ocp-schedules-14,e6076f44464de792,bfc3ef0835432dbb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:10:41.676 INFO 4486 --- [ocp-async-4,9cde9a70e3d22903,7091abc609d9ccb5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:42.216 INFO 4486 --- [ocp-schedules-27,375e66f98605c1cc,835fa0b5c0013da7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2024-10-16 14:10:42.281 INFO 4486 --- [ocp-schedules-11,c0c3955c2ae32d1f,5e8f350fb6e8b8a2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2024-10-16 14:10:42.283 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1ca2be0773621924,17c8a090cfe9fd71] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=1ca2be0773621924, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:42.283 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-2,1ca2be0773621924,17c8a090cfe9fd71] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=1ca2be0773621924, duration=2 ms] 2024-10-16 14:10:42.308 WARN 4486 --- [scanner-13342979-1,c0c3955c2ae32d1f,178a81a77985cefc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 55 2024-10-16 14:10:42.361 INFO 4486 --- [alarm-detect-4,c0c3955c2ae32d1f,f05d86d289246daf] c.o.o.alarm.core.detect.MetricDetector : alarmType host_ntp_service_not_exist detect time range 1729059015 ~ 1729059032 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ntp_service_exist=[SeriesMetricValues(metric=Metric{name='ntp_service_exist', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729059030, value=1.0)])], ntp_service_available=[SeriesMetricValues(metric=Metric{name='ntp_service_available', labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176}}, data=[ValueData(timestamp=1729059030, value=0.0)])]}, metricEvaluateResults={ntp_service_exist == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_exist, match=false, value=1.0, values={}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})], ntp_service_available == 0=[ExpressionEvaluateResult(matchValue=0.0, metric=ntp_service_available, match=true, value=0.0, values={1729059030=0.0}, valueCount=1, labels={app=HOST, env_type=host, host=10.197.49.176, host_name=yuj-ob176, ob_cluster_id=1701313639, ob_cluster_name=obce_3zones, obzone=zone1, svr_ip=10.197.49.176})]}, matched=true, isContainValidValue=null, message=null, level=2) 2024-10-16 14:10:42.568 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b2ab182bfd6f012f,84e824849adc4561] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=b2ab182bfd6f012f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:42.568 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-10,b2ab182bfd6f012f,84e824849adc4561] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=b2ab182bfd6f012f, duration=1 ms] 2024-10-16 14:10:42.858 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b7e8a70ffc0acd03,a963aed00ea2900e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obce_3zones&VersionOnly=true, client=10.197.49.177, traceId=b7e8a70ffc0acd03, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obce_3zones,true,] 2024-10-16 14:10:42.859 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-4,b7e8a70ffc0acd03,a963aed00ea2900e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.177, traceId=b7e8a70ffc0acd03, duration=2 ms] 2024-10-16 14:10:43.041 INFO 4486 --- [ocp-async-9,d0a1f000c0fbb5e8,48c350d4fcfabc15] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2024-10-16 14:10:43.608 INFO 4486 --- [ocp-schedules-9,bcba2a2ac171a13f,dd9b051a5a290b98] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:10:43.676 INFO 4486 --- [ocp-async-5,95a32652dbb16d9e,9b351c4e13a3a6a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:43.707 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:43.994 INFO 4486 --- [prometheus-minute-schedule-1,f8d3825d05fac85d,fd5608b8b718ed23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:44.001 INFO 4486 --- [prometheus-minute-schedule-1,f8d3825d05fac85d,fd5608b8b718ed23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:45.664 INFO 4486 --- [ocp-schedules-26,25d1b821448f0ceb,27b0f0d56942800e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2024-10-16 14:10:45.669 INFO 4486 --- [ocp-schedules-26,25d1b821448f0ceb,27b0f0d56942800e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2024-10-16 14:10:45.669 INFO 4486 --- [ocp-schedules-26,25d1b821448f0ceb,27b0f0d56942800e] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 3 2024-10-16 14:10:45.677 INFO 4486 --- [ocp-async-9,ab5db8e0617c7780,6f5a9c0fea6f7020] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:45.682 INFO 4486 --- [ocp-schedules-9,961ba116d9d82605,9b5c98f200d37f2f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2024-10-16 14:10:45.682 INFO 4486 --- [ocp-schedules-9,961ba116d9d82605,9b5c98f200d37f2f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2024-10-16 14:10:46.708 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:46.994 INFO 4486 --- [prometheus-minute-schedule-1,d822ddd1aefe93b5,264419154f5ecd4a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:47.000 INFO 4486 --- [prometheus-minute-schedule-1,d822ddd1aefe93b5,264419154f5ecd4a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:47.071 INFO 4486 --- [ocp-schedules-6,b7b304d9bb787063,c8ddbcf6c105e4dc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=3, successCount=3 2024-10-16 14:10:47.203 INFO 4486 --- [ocp-schedules-4,023ea23cf6f7799d,f207e44dfdb18783] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=3, successCount=3 2024-10-16 14:10:47.339 INFO 4486 --- [ocp-schedules-13,8d99994392859038,64b8bfbd21bb3634] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2024-10-16 14:10:47.677 INFO 4486 --- [ocp-async-3,0aa6444d6a9863ed,145bf8d53528c178] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:48.739 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,6b8826e090059e24,546be9bf80599195] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=6b8826e090059e24, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:48.740 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-3,6b8826e090059e24,546be9bf80599195] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=6b8826e090059e24, duration=2 ms] 2024-10-16 14:10:49.677 INFO 4486 --- [ocp-async-2,b649da78fc8d4c13,728e30d5ba8bfc1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2024-10-16 14:10:49.708 INFO 4486 --- [pool-12-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : After refresh, the leader address is :10.197.49.180:8080 2024-10-16 14:10:49.994 INFO 4486 --- [prometheus-minute-schedule-1,83bd6b82cd690b4b,33db1bc1c6f00712] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1729059060, interval=60 2024-10-16 14:10:50.001 INFO 4486 --- [prometheus-minute-schedule-1,83bd6b82cd690b4b,33db1bc1c6f00712] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=16, collectTaskCount=16 2024-10-16 14:10:50.628 INFO 4486 --- [ocp-schedules-25,4283863874f31b15,a34afc2671bee21c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2024-10-16 14:10:50.628 INFO 4486 --- [ocp-schedules-25,4283863874f31b15,a34afc2671bee21c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=700500076117168128 2024-10-16 14:10:50.629 INFO 4486 --- [ocp-schedules-25,4283863874f31b15,a34afc2671bee21c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2024-10-16 14:10:50.629 INFO 4486 --- [ocp-schedules-25,4283863874f31b15,a34afc2671bee21c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=701258284087709696 2024-10-16 14:10:50.630 INFO 4486 --- [ocp-schedules-25,4283863874f31b15,a34afc2671bee21c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2024-10-16 14:10:50.630 INFO 4486 --- [ocp-schedules-25,4283863874f31b15,a34afc2671bee21c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=703046246634950656 2024-10-16 14:10:50.795 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fb3e18e68538e0ea,f8db0e15dc56d97c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=OMS&UID=OMS&ObRegion=obce_3zones, client=10.197.49.179, traceId=fb3e18e68538e0ea, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=obce_3zones,false,0,] 2024-10-16 14:10:50.795 INFO 4486 --- [http-nio-0.0.0.0-8080-exec-6,fb3e18e68538e0ea,f8db0e15dc56d97c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.197.49.179, traceId=fb3e18e68538e0ea, duration=1 ms] 2024-10-16 14:10:51.622 INFO 4486 --- [ocp-schedules-29,878c371e251e9e2b,c9621821c51f2ed7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2024-10-16 14:10:51.678 INFO 4486 --- [ocp-async-10,ac83daa4ae18c3b3,a820de2b3f4e76dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents